解析不同的日期格式：正则表达式
Parse Different Date formats: Regex
重新发布这个问题的细节（因为最后一个被标记下来）。
我正在努力从档案卡中解析杂乱无章的 (tessearct-ocr) 以获得至少 50% 的信息 (date1)。根据下面的数据示例，数据行包含不同形式的日期。
Raw_Text
1 "15957-8 . 3n v g - vw, 1 ekresta . bowker, william e tley n0 .qu v- l. c.
s. peteris, forestville, n. y. .mafae date1 june 17,1942 by davis, c. j6
l. g. b. jonnis, buffalo, n. y. ngsted decl 17, 1949.3y 7 davis, c. j.
date3 by j date4 - by date5 by 6 -.5/, 7/19/l date6 17 jul 1916 salamanca.
hf date7 31 dec 1986 buffalo, new york "
2 ".1o2o83n5ddn.. -i ekresta i bowles, albert edwin i made date1 june 9p1909
by parker, elm. date2 dec . 18 w date3 . by dep osed by date5 by date7mqm
9 ivvld wm 4144, mac, .75 076 eaqlwli "
3 "i naime bowles, charles edward made date1 may 31. 1892 by mclaren, wneoi
date2 may 18. 1895 by mclaren, w.e. date3 . i by date4 may 10. 1908 by
bip. of chicago. date5 by date7 "
4 "101 557 am l i ekrestaibowles, donald manson ..46 ohio trlnlty cathedral,
cleveland, ohio made date1 6/19/76 by burt, ji. h. grace , cleveland, ohio
date2 11 jun 77 by bp j h burt date3 . 1 .. by date4 by date5 bv m cuyahoga
heights, ohio date6 4/29/27 date7 240000 "
5 "227354 101 575 m68, frederick augustus st. paujjs cathedral, buffalo,
n.y. made date1 6/15/63 by scaife. l.i... st. thomas. modia, bath, n.y.
date2 1/11/611 by scaife. l.eo date3 by date4 by date5 by bradford, n.y. i
. 130m 6/1/18 date7 17 jun 1996 foratvme new york z4uc-xl "
6 "1 95812d ll. il ekresta bowles, harry oscar lmade date14 july 17, 190433,
lepnard, w.a. date2 july 25 , 1905 by leonard, w.a. i date3 by date4 by
date5 by g- m. /(,,/mr date7 jay /z/,. /357i l /mwi yk/maj. "
7 "5025 ,.. 2.57631 il . - . .. .1 i ekresta bowles , jedwiah hibbafd made
deac0n 8., i5-0i1862i13y potter, iih. date2 10. 280 1864 1 biy stevens, w.
b. date3 by date4 7 .30 l 1875 by date5 by date7 "
8 "30.611126 ekhq il ekresta bowles, ralph hart made date1 12. 210 i1883 by
iwiiiliams, i36 date2 7.. 1. 1885 by williams , j. date3 by i date4 by
date5 by g .97) l/am 9- date7 10. 4. 1900 (78) if x/ma 3.4, 154.47.11.73.
4,... mya-ix "
9 "2.25678 . 1o14593 ekresta bowles, robert brigham, jr. st. matthew s
cathedra1,da11quexas made date1 6/18/65 by mason, c. a. 57 mmzws camp
dr7///9s tams date2 12 21 cs by 14.45.42 c a date3 i by date4 by date5 ,
by houston, texas date6 4/11/30 date7 12 dec 2000 dallas texas 2400-xi "
10 "101 619 34hq woe ekresta bowlin1 howard bruce cathedral modia of saint
peter 61 st. paul, washin ton, dc made date1 13 jun 92 bybp r h haines
(wdc st. alban1s modia, annandale, vir inia . pdumd 16 jan 93 by r h halnes
(wdc) date3 by atas by date4 v by date5 by date6 31 aug 1946 e st. louis.
il date7 2400-i "
11 "w k8 8km tm boiling jack dnnmwm q- f grace ch , made dat j 11201). salem
mares. stverrett. f. ,w a x st. johms modia. memphis, tenh. date1 apr. 25.
1955 - bv barth, t.in.. date3 4 by date4 by date5 by date7 wq iw r 1 w .n
. 4.1- 1 date6z1l7i1c. "
我通过两步过程解析 date1，
- 1.解析名称"date1"和"by"之间的文本
- 2.使用日期解析器提取实际日期
import re
import dateutil.parser as dparser
for lines in Raw_Text:
lines = lines.lower() #make lower case
lines = lines.strip() #remove leading and ending spaces
lines = " ".join(lines.split()) #remove duplicated spaces
# Step 1
#Extract data between "date1" and "by"
deacondt = re.findall(r'date1(.*?)by',lines)
deacondt = ''.join(deacondt) #Convert list to a string
# Step 2
# use dateutil to parse dates in extracted data
try:
deacondt1 = dparser.parse(deacondt)
except:
deacondt1 = 'NA'
print deacondt1
步骤 1 的输出是，
[' june 17,1942 ']
[' june 9p1909 ']
[' may 31. 1892 ']
[' 6/19/76 ']
[' 6/15/63 ']
['4 july 17, 190433, lepnard, w.a. date2 july 25 , 1905 ']
[]
[' 12. 210 i1883 ']
[' 6/18/65 ']
[' 13 jun 92 ']
[]
而第2步returns输出如下
2018-06-17 00:00:00
1909-06-17 21:00:00
1892-05-31 00:00:00
1976-06-19 00:00:00
2063-06-15 00:00:00
NA
NA
NA
2065-06-18 00:00:00
1992-06-13 00:00:00
NA
第 2 步未能提供所有日期。 Python 2.7 是否有比 "dateutil.parser" 更好的日期解析器？
你的许多日期都有不同的格式：这会让事情变得困难。
您可以使用 datetime 库来解析日期。由于您的数据有多种格式，因此您将需要几种不同的格式字符串。
datetime有两个有用的函数：datetime.strptime（string PARSE时间，returns datetime.datetime）和datetime.strftime（string FROM时间，returns str)
这是一个如何解析的示例，前提是您有足够的格式字符串：
import datetime
for lines in Raw_Text:
## Do the regex stuff above.
## Keep each returned result as a separate string.
regex_results = get_your_regex_results()
# Step 2
# use dateutil to parse dates in extracted data
date_formats = [ ## You will need several formats to try.
'%m/%d/%Y',
]
for datestring in regex_results:
for fmt in date_formats:
try:
date_str = date_str.strip()
deacondt1 = datetime.datetime.strptime(date_str, fmt)
print(deacondt1)
break
except ValueError:
continue
https://docs.python.org/2/library/datetime.html#strftime-strptime-behavior
你可以试试这个，
deacondt1 = dparser.parse(deacondt, dayfirst=False, fuzzy=True)
fuzzy – 允许包含 un-dateformat 个单词的字符串，例如“Today is January 1, 2047 at 8:21:00AM”。dayfirst=False 表示 month-first date-format 输入像您一样的字符串。
但是 dateutil-parser 不足以提取您想要的输出，因此需要将更接近日期格式的字符串传递给解析器。
Regex 提取关于 date1
的字符串
(?s)date1\d?((?:(?!by|date2|date3).)*)
Demo,其中不仅'by'而且'date2'和'date3'都用作separator和date10~date19被视为date1.
然后，对提取的字符串进行操作（删除前导和尾随空格等）以获得 date-util 解析器可接受的输入。
regx= re.compile(r'(?s)date1\d?((?:(?!by|date2|date3).)*)')
raw_date= [re.sub(r'(?i)(?<=\s)[a-z]?(\d{4}|\d{2})\d*', r'', re.sub(r'\s+|,|(?<=\d)[^\d\s\/](?=\d)',' ', re.sub(r'^\s+|\s+$|\n+','', m))) for m in regx.findall(Raw_Text)]
for deacondt in raw_date:
try:
deacondt1 = dparser.parse(deacondt, dayfirst=False, fuzzy=True)
except:
deacondt1 = 'NA'
print(deacondt +"\n"+ str(deacondt1))
输出
june 17 1942
1942-06-17 00:00:00
june 9 1909
1909-06-09 00:00:00
may 31. 1892
1892-05-31 00:00:00
6/19/76
1976-06-19 00:00:00
6/15/63
2063-06-15 00:00:00
july 17 1904 lepnard w.a.
1904-07-17 00:00:00
12. 21 1883
1883-12-21 00:00:00
6/18/65
2065-06-18 00:00:00
13 jun 92
1992-06-13 00:00:00
apr. 25. 1955 - bv barth t.in..
1955-04-25 00:00:00
没有解析模块可以为您可能遇到的每个 OCR 曲线提供完整的解决方案。
您将不得不构建一些 evaluation/correction 框架来发现并修复您可以修复的问题。
我建议采用以下工作流程：
- 尝试解析日期序列。
- 将未解析的序列保存到特殊文件中
- 编辑文件，添加一些正则表达式替换规则以将序列重写为可挽救的形式。
- 应用文件中的规则并再次尝试解析
- 从 2. 开始重复，直到处理完所有内容。
这是一些示例代码：
parser.py
import re
import csv
import glob, os
from datetime import datetime
import dateutil.parser as dparser
def load_patterns():
''' load patterns from existing pat_*.csv
return a dict of the form { sequence: [sequence,pattern,replace] }
sequence is an example of the string that should be handled by this pattern
pattern and replace have the same meaning as for re.sub
'''
patterns = {}
for pattern_file in glob.glob("pat_*.csv"):
with open(pattern_file, 'r') as fh:
reader = csv.DictReader(fh, delimiter=',', quotechar='"', skipinitialspace=True)
reader.fieldnames=[f.strip() for f in reader.fieldnames]
for row in reader:
# skipping empty patterns if there was non-empty one for this sequence
if row['sequence'] in patterns and not row['pattern']:
continue
patterns[row['sequence']]=(row['pattern'],row['replace'])
return patterns
def save_nonmatched(patterns, nonmatched):
''' saves a new pattern file with the empty pattern field
supposed to be edited manually afterwards
'''
items_to_save = [ key for key in nonmatched if key not in patterns ]
if not items_to_save:
return
new_file=datetime.now().strftime('pat_%Y%m%d_%H%M%S.csv')
with open(new_file, 'w', newline='') as fh:
writer = csv.DictWriter(fh, fieldnames=['sequence', 'pattern', 'replace'], quoting=csv.QUOTE_ALL)
writer.writeheader()
for key in items_to_save:
writer.writerow({'sequence':key, 'pattern':'', 'replace':''})
def sub_with_patterns(s, patterns):
''' try to match each pattern in patterns iterable
return expanded string if match succeeded
'''
debug=1
for sequence, (pattern, replace) in patterns.items():
if not pattern:
continue
match=re.search(pattern,s,re.X)
if match:
return match.expand(replace)
return None
nomatch={}
patterns = load_patterns()
Raw_Text = re.sub(r'\s+', ' ' ,open('in.txt','r').read().lower()).strip()
for dt in re.findall(r'date1(.*?)by', Raw_Text, re.S):
corrected = sub_with_patterns(dt, patterns)
try:
parsed = dparser.parse(corrected or dt)
print ("input:%s parsed:%s" % (dt,parsed))
except:
nomatch[dt]=1
print ("input:%s ** not parsed" % (dt))
save_nonmatched(patterns, nomatch)
现在，如果在您的输入上尝试脚本，我们会得到第一个更正 csv：
"sequence","pattern","replace"
"4 july 17, 190433, lepnard, w.a. date2 july 25 , 1905 ","",""
" 12. 210 i1883 ","",""
" apr. 25. 1955 - bv barth, t.in.. date3 4 ","",""
和输出：
input: june 17,1942 parsed:2018-06-17 00:00:00
...
input:4 july 17, 190433, lepnard, w.a. date2 july 25 , 1905 ** not parsed
...
我们编辑文件如下：
"sequence","pattern","replace"
"4 july 17, 190433, lepnard, w.a. date2 july 25 , 1905 ","^
\s*(?P<day>\d+)
\s+(?P<month>(?:jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)[a-z]*)
\s+(?P<year>\d{2})
","\g<day> \g<month> 19\g<year>"
" 12. 210 i1883 ","",""
" apr. 25. 1955 - bv barth, t.in.. date3 4 ","",""
并且 运行 解析器再次：
input: june 17,1942 parsed:2018-06-17 00:00:00
...
input:4 july 17, 190433, lepnard, w.a. date2 july 25 , 1905 parsed:1917-07-04 00:00:00
...
当然，这远远不能解决您将要遇到的所有 OCR 解析问题，但这可能是一个好的开始。
重新发布这个问题的细节（因为最后一个被标记下来）。
我正在努力从档案卡中解析杂乱无章的 (tessearct-ocr) 以获得至少 50% 的信息 (date1)。根据下面的数据示例，数据行包含不同形式的日期。
Raw_Text
1 "15957-8 . 3n v g - vw, 1 ekresta . bowker, william e tley n0 .qu v- l. c.
s. peteris, forestville, n. y. .mafae date1 june 17,1942 by davis, c. j6
l. g. b. jonnis, buffalo, n. y. ngsted decl 17, 1949.3y 7 davis, c. j.
date3 by j date4 - by date5 by 6 -.5/, 7/19/l date6 17 jul 1916 salamanca.
hf date7 31 dec 1986 buffalo, new york "
2 ".1o2o83n5ddn.. -i ekresta i bowles, albert edwin i made date1 june 9p1909
by parker, elm. date2 dec . 18 w date3 . by dep osed by date5 by date7mqm
9 ivvld wm 4144, mac, .75 076 eaqlwli "
3 "i naime bowles, charles edward made date1 may 31. 1892 by mclaren, wneoi
date2 may 18. 1895 by mclaren, w.e. date3 . i by date4 may 10. 1908 by
bip. of chicago. date5 by date7 "
4 "101 557 am l i ekrestaibowles, donald manson ..46 ohio trlnlty cathedral,
cleveland, ohio made date1 6/19/76 by burt, ji. h. grace , cleveland, ohio
date2 11 jun 77 by bp j h burt date3 . 1 .. by date4 by date5 bv m cuyahoga
heights, ohio date6 4/29/27 date7 240000 "
5 "227354 101 575 m68, frederick augustus st. paujjs cathedral, buffalo,
n.y. made date1 6/15/63 by scaife. l.i... st. thomas. modia, bath, n.y.
date2 1/11/611 by scaife. l.eo date3 by date4 by date5 by bradford, n.y. i
. 130m 6/1/18 date7 17 jun 1996 foratvme new york z4uc-xl "
6 "1 95812d ll. il ekresta bowles, harry oscar lmade date14 july 17, 190433,
lepnard, w.a. date2 july 25 , 1905 by leonard, w.a. i date3 by date4 by
date5 by g- m. /(,,/mr date7 jay /z/,. /357i l /mwi yk/maj. "
7 "5025 ,.. 2.57631 il . - . .. .1 i ekresta bowles , jedwiah hibbafd made
deac0n 8., i5-0i1862i13y potter, iih. date2 10. 280 1864 1 biy stevens, w.
b. date3 by date4 7 .30 l 1875 by date5 by date7 "
8 "30.611126 ekhq il ekresta bowles, ralph hart made date1 12. 210 i1883 by
iwiiiliams, i36 date2 7.. 1. 1885 by williams , j. date3 by i date4 by
date5 by g .97) l/am 9- date7 10. 4. 1900 (78) if x/ma 3.4, 154.47.11.73.
4,... mya-ix "
9 "2.25678 . 1o14593 ekresta bowles, robert brigham, jr. st. matthew s
cathedra1,da11quexas made date1 6/18/65 by mason, c. a. 57 mmzws camp
dr7///9s tams date2 12 21 cs by 14.45.42 c a date3 i by date4 by date5 ,
by houston, texas date6 4/11/30 date7 12 dec 2000 dallas texas 2400-xi "
10 "101 619 34hq woe ekresta bowlin1 howard bruce cathedral modia of saint
peter 61 st. paul, washin ton, dc made date1 13 jun 92 bybp r h haines
(wdc st. alban1s modia, annandale, vir inia . pdumd 16 jan 93 by r h halnes
(wdc) date3 by atas by date4 v by date5 by date6 31 aug 1946 e st. louis.
il date7 2400-i "
11 "w k8 8km tm boiling jack dnnmwm q- f grace ch , made dat j 11201). salem
mares. stverrett. f. ,w a x st. johms modia. memphis, tenh. date1 apr. 25.
1955 - bv barth, t.in.. date3 4 by date4 by date5 by date7 wq iw r 1 w .n
. 4.1- 1 date6z1l7i1c. "
我通过两步过程解析 date1， - 1.解析名称"date1"和"by"之间的文本 - 2.使用日期解析器提取实际日期
import re
import dateutil.parser as dparser
for lines in Raw_Text:
lines = lines.lower() #make lower case
lines = lines.strip() #remove leading and ending spaces
lines = " ".join(lines.split()) #remove duplicated spaces
# Step 1
#Extract data between "date1" and "by"
deacondt = re.findall(r'date1(.*?)by',lines)
deacondt = ''.join(deacondt) #Convert list to a string
# Step 2
# use dateutil to parse dates in extracted data
try:
deacondt1 = dparser.parse(deacondt)
except:
deacondt1 = 'NA'
print deacondt1
步骤 1 的输出是，
[' june 17,1942 ']
[' june 9p1909 ']
[' may 31. 1892 ']
[' 6/19/76 ']
[' 6/15/63 ']
['4 july 17, 190433, lepnard, w.a. date2 july 25 , 1905 ']
[]
[' 12. 210 i1883 ']
[' 6/18/65 ']
[' 13 jun 92 ']
[]
而第2步returns输出如下
2018-06-17 00:00:00
1909-06-17 21:00:00
1892-05-31 00:00:00
1976-06-19 00:00:00
2063-06-15 00:00:00
NA
NA
NA
2065-06-18 00:00:00
1992-06-13 00:00:00
NA
第 2 步未能提供所有日期。 Python 2.7 是否有比 "dateutil.parser" 更好的日期解析器？
你的许多日期都有不同的格式：这会让事情变得困难。
您可以使用 datetime 库来解析日期。由于您的数据有多种格式，因此您将需要几种不同的格式字符串。
datetime有两个有用的函数：datetime.strptime（string PARSE时间，returns datetime.datetime）和datetime.strftime（string FROM时间，returns str)
这是一个如何解析的示例，前提是您有足够的格式字符串：
import datetime
for lines in Raw_Text:
## Do the regex stuff above.
## Keep each returned result as a separate string.
regex_results = get_your_regex_results()
# Step 2
# use dateutil to parse dates in extracted data
date_formats = [ ## You will need several formats to try.
'%m/%d/%Y',
]
for datestring in regex_results:
for fmt in date_formats:
try:
date_str = date_str.strip()
deacondt1 = datetime.datetime.strptime(date_str, fmt)
print(deacondt1)
break
except ValueError:
continue
https://docs.python.org/2/library/datetime.html#strftime-strptime-behavior
你可以试试这个，
deacondt1 = dparser.parse(deacondt, dayfirst=False, fuzzy=True)
fuzzy– 允许包含un-dateformat个单词的字符串，例如“Today is January 1, 2047 at 8:21:00AM”。dayfirst=False表示month-first date-format输入像您一样的字符串。
但是 dateutil-parser 不足以提取您想要的输出，因此需要将更接近日期格式的字符串传递给解析器。
Regex 提取关于 date1
(?s)date1\d?((?:(?!by|date2|date3).)*)
Demo,其中不仅'by'而且'date2'和'date3'都用作separator和date10~date19被视为date1.
然后，对提取的字符串进行操作（删除前导和尾随空格等）以获得 date-util 解析器可接受的输入。
regx= re.compile(r'(?s)date1\d?((?:(?!by|date2|date3).)*)')
raw_date= [re.sub(r'(?i)(?<=\s)[a-z]?(\d{4}|\d{2})\d*', r'', re.sub(r'\s+|,|(?<=\d)[^\d\s\/](?=\d)',' ', re.sub(r'^\s+|\s+$|\n+','', m))) for m in regx.findall(Raw_Text)]
for deacondt in raw_date:
try:
deacondt1 = dparser.parse(deacondt, dayfirst=False, fuzzy=True)
except:
deacondt1 = 'NA'
print(deacondt +"\n"+ str(deacondt1))
输出
june 17 1942
1942-06-17 00:00:00
june 9 1909
1909-06-09 00:00:00
may 31. 1892
1892-05-31 00:00:00
6/19/76
1976-06-19 00:00:00
6/15/63
2063-06-15 00:00:00
july 17 1904 lepnard w.a.
1904-07-17 00:00:00
12. 21 1883
1883-12-21 00:00:00
6/18/65
2065-06-18 00:00:00
13 jun 92
1992-06-13 00:00:00
apr. 25. 1955 - bv barth t.in..
1955-04-25 00:00:00
没有解析模块可以为您可能遇到的每个 OCR 曲线提供完整的解决方案。
您将不得不构建一些 evaluation/correction 框架来发现并修复您可以修复的问题。
我建议采用以下工作流程：
- 尝试解析日期序列。
- 将未解析的序列保存到特殊文件中
- 编辑文件，添加一些正则表达式替换规则以将序列重写为可挽救的形式。
- 应用文件中的规则并再次尝试解析
- 从 2. 开始重复，直到处理完所有内容。
这是一些示例代码：
parser.py
import re
import csv
import glob, os
from datetime import datetime
import dateutil.parser as dparser
def load_patterns():
''' load patterns from existing pat_*.csv
return a dict of the form { sequence: [sequence,pattern,replace] }
sequence is an example of the string that should be handled by this pattern
pattern and replace have the same meaning as for re.sub
'''
patterns = {}
for pattern_file in glob.glob("pat_*.csv"):
with open(pattern_file, 'r') as fh:
reader = csv.DictReader(fh, delimiter=',', quotechar='"', skipinitialspace=True)
reader.fieldnames=[f.strip() for f in reader.fieldnames]
for row in reader:
# skipping empty patterns if there was non-empty one for this sequence
if row['sequence'] in patterns and not row['pattern']:
continue
patterns[row['sequence']]=(row['pattern'],row['replace'])
return patterns
def save_nonmatched(patterns, nonmatched):
''' saves a new pattern file with the empty pattern field
supposed to be edited manually afterwards
'''
items_to_save = [ key for key in nonmatched if key not in patterns ]
if not items_to_save:
return
new_file=datetime.now().strftime('pat_%Y%m%d_%H%M%S.csv')
with open(new_file, 'w', newline='') as fh:
writer = csv.DictWriter(fh, fieldnames=['sequence', 'pattern', 'replace'], quoting=csv.QUOTE_ALL)
writer.writeheader()
for key in items_to_save:
writer.writerow({'sequence':key, 'pattern':'', 'replace':''})
def sub_with_patterns(s, patterns):
''' try to match each pattern in patterns iterable
return expanded string if match succeeded
'''
debug=1
for sequence, (pattern, replace) in patterns.items():
if not pattern:
continue
match=re.search(pattern,s,re.X)
if match:
return match.expand(replace)
return None
nomatch={}
patterns = load_patterns()
Raw_Text = re.sub(r'\s+', ' ' ,open('in.txt','r').read().lower()).strip()
for dt in re.findall(r'date1(.*?)by', Raw_Text, re.S):
corrected = sub_with_patterns(dt, patterns)
try:
parsed = dparser.parse(corrected or dt)
print ("input:%s parsed:%s" % (dt,parsed))
except:
nomatch[dt]=1
print ("input:%s ** not parsed" % (dt))
save_nonmatched(patterns, nomatch)
现在，如果在您的输入上尝试脚本，我们会得到第一个更正 csv：
"sequence","pattern","replace"
"4 july 17, 190433, lepnard, w.a. date2 july 25 , 1905 ","",""
" 12. 210 i1883 ","",""
" apr. 25. 1955 - bv barth, t.in.. date3 4 ","",""
和输出：
input: june 17,1942 parsed:2018-06-17 00:00:00
...
input:4 july 17, 190433, lepnard, w.a. date2 july 25 , 1905 ** not parsed
...
我们编辑文件如下：
"sequence","pattern","replace"
"4 july 17, 190433, lepnard, w.a. date2 july 25 , 1905 ","^
\s*(?P<day>\d+)
\s+(?P<month>(?:jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)[a-z]*)
\s+(?P<year>\d{2})
","\g<day> \g<month> 19\g<year>"
" 12. 210 i1883 ","",""
" apr. 25. 1955 - bv barth, t.in.. date3 4 ","",""
并且 运行 解析器再次：
input: june 17,1942 parsed:2018-06-17 00:00:00
...
input:4 july 17, 190433, lepnard, w.a. date2 july 25 , 1905 parsed:1917-07-04 00:00:00
...
当然，这远远不能解决您将要遇到的所有 OCR 解析问题，但这可能是一个好的开始。