转换没有分隔符的平面文件

Question

我想将平面文件 test.txt 转换为平面文件 test-output.txt.

方案如下：

示例输入：test.txt

COD/ID:37
PRJ/NAME: Josephy Murphy
PRJ/EMAIL: jmurphy@email.com
PRJ/DESCRIPTION: test37, test37, test37 ...

COD/ID:38
PRJ/NAME: Paul Newman
PRJ/EMAIL: pnewman@email.com
PRJ/DESCRIPTION: test38, test38, test38 ...

.
.

示例输出：test-output.txt（竖线分隔，无标签）

37|Josephy Murphy|jmurphy@email.com|test37, test37, test37 ...
38|Paul Newman|pnewman@email.com|test38, test38, test38 ...
.
.

截图链接：
test.txt
test-output.txt

我想将此文件导入 SQL 服务器。但是文件 test.txt（15,000,000 行）默认情况下不带分隔符导入。

我将使用SSIS导入数据，但必须是CSV格式或其他带分隔符的格式。

我考虑过使用 REGEX 或 SSIS 脚本组件。我知道带格式文本的 SSIS 文件的导入过程，但是这个文件没有格式化。

Answer 1

以正则表达式为例：

    class Program
    {
        private static Regex reg = new Regex(@"COD/ID:\s(?<id>\d+)\r\nPRJ/NAME:\s(?<name>.+?)\r\nPRJ/EMAIL:\s(?<email>\S+?@\S+?\.\S+?)\r\nPRJ/DESCRIPTION:\s(?<description>.*?)(?:\n|$)");

        static void Main(string[] args)
        {
            string original = @"
COD/ID: 37
PRJ/NAME: Josephy Murphy
PRJ/EMAIL: jmurphy@email.com
PRJ/DESCRIPTION: test37, test37, test37 ...

COD/ID: 38
PRJ/NAME: Paul Newman
PRJ/EMAIL: pnewman@email.com
PRJ/DESCRIPTION: test38, test38, test38 ...";


            string result = string.Join(
                "\n",
                reg.Matches(original)
                .Cast<Match>()
                .Select(m => string.Format("{0}|{1}|{2}|{3}",m.Groups["id"].Value,m.Groups["name"].Value,m.Groups["email"].Value,m.Groups["description"].Value)));
            Console.WriteLine(result);
        }
    }

编辑

class Program
{
    private static Regex reg = new Regex(@"COD/ID:\s(?<id>\d+)\r\nPRJ/NAME:\s(?<name>.+?)\r\nPRJ/EMAIL:\s(?<email>\S+?@\S+?\.\S+?)\r\nPRJ/DESCRIPTION:\s(?<description>.*?)\r\n");

    static void Main(string[] args)
    {
        StringBuilder intermediateStringBuilder = new StringBuilder();

        using (StreamReader reader = new StreamReader(@"YourInputPath.txt",true))
        {               
            using (StreamWriter writer = new StreamWriter("YourOutputPath.txt"))
            {
                while (reader.Peek() > 0)
                {
                    string line = reader.ReadLine();
                    if (!string.IsNullOrWhiteSpace(line))
                    {
                        intermediateStringBuilder.AppendLine(line);
                    }
                    else
                    {
                        WriteToFile(intermediateStringBuilder, writer);
                    }
                } 
                WriteToFile(intermediateStringBuilder,writer);
            }
        }
    }

    private static void WriteToFile(StringBuilder intermediateStringBuilder, StreamWriter writer)
    {
        Match m = reg.Match(intermediateStringBuilder.ToString());
        writer.WriteLine("{0}|{1}|{2}|{3}", m.Groups["id"].Value, m.Groups["name"].Value, m.Groups["email"].Value, m.Groups["description"].Value);
        intermediateStringBuilder.Clear();
    }
}

Answer 2

在这种情况下，您可以在没有正则表达式的情况下执行此操作，因为上下文是已知的。

使用这个：

public class EntryN
{
   public string id { get; set; }
   public string name { get; set; }
   public string email { get; set; }
   public string description { get; set; }

   public EntryN()
   {
      this.id = this.name = this.email = this.description = string.Empty;
   }
   public string ToLine()
   { 
       return this.id + "|" + this.name + "|" + this.email + "|" + this.description; 
   }
}

var entries = new List<EntryN>();
using (var sl = new StreamReader(@"c:\YOURPATH.txt", true))
{
    var entry = new EntryN();
    var line = string.Empty;
    while ((line = sl.ReadLine()) != null)
    {
       if (line.StartsWith("COD/ID:"))
          entry.id = line.Substring(8).Trim();
       else if (line.StartsWith("PRJ/NAME:"))
          entry.name = line.Substring(10).Trim();
       else if (line.StartsWith("PRJ/EMAIL"))
          entry.email = line.Substring(11).Trim();
       else if (line.StartsWith("PRJ/DESCRIPTION"))
          entry.description = line.Substring(17).Trim();
      else if (line.Trim() == string.Empty)
      {
          entries.Add(entry);
          entry = new EntryN();
      }
    }
    if (!entry.Equals(new EntryN()))
       entries.Add(entry);
    sl.Close();
}

var resulted = entries.Select(p => p.ToLine()).ToList();

输出：

编辑：另一个没有单独 class 的代码将直接写入而不创建额外的字符串：

var id = string.Empty;
var name = string.Empty;
var email = string.Empty;
var description = string.Empty;
using (var sw = new StreamWriter(@"OUTPUT_FILE", false, Encoding.UTF8))
{
    using (var sl = new StreamReader(@"INPUT_FILE", true))
    {
       var line = string.Empty;
       while ((line = sl.ReadLine()) != null)
       {
           if (line.StartsWith("COD/ID:"))
              id = line.Substring(8).Trim();
           else if (line.StartsWith("PRJ/NAME:"))
              name = line.Substring(10).Trim();
           else if (line.StartsWith("PRJ/EMAIL"))
              email = line.Substring(11).Trim();
           else if (line.StartsWith("PRJ/DESCRIPTION"))
              description = line.Substring(17).Trim();
           else if (line.Trim() == string.Empty)
           {
               sw.WriteLine(string.Format("{0}|{1}|{2}|{3}", id, name, email, description));
               id = name = email = description = string.Empty;
            }
        }
        if (!new string[] {id, name, email, description}.Any(p => string.IsNullOrWhiteSpace(p)))
            sw.WriteLine(string.Format("{0}|{1}|{2}|{3}", id, name, email, description));
        sl.Close();
     }
     sw.Close();
 }

转换没有分隔符的平面文件

Transform Flat File without delimiters

c#

regex

text

ssis

方案如下：