如何从一个巨大的单行文本文件中提取 URL?

How can I extract URLs from a huge one line text file?

我有一个文本文件,我想从中提取链接。

问题是文本文件只有一行,链接很多!

或者当我在记事本中打开它时,它显示在很多文件中但没有组织。

示例文本:

[{"participants": ["minanageh379", "xcsadc"], "conversation": [{"sender": "minanageh379", "created_at": "2019-04-12T12:51:56.560361+00:00", "media": "https://scontent-lax3-1.cdninstagram.com/vp/edddf95178aca7bf75930ab8698ee45b/5D45203B/t51.2885-15/fr/e15/s1080x1080/55823673_114448266206459_7321604432125975069_n.jpg?_nc_ht=scontent-lax3-1.cdninstagram.com&ig_cache_key=Mjg2ODYwNDMxNzU3OTI1MTE1NTAxNjQ1NTk5MDkwOTMzNzY%3D.2"}, {"sender": "minanageh379", "created_at": "2019-04-12T12:51:51.923138+00:00", "text": "sd"}, {"sender": "minanageh379", "created_at": "2019-04-12T12:51:41.689524+00:00", "text": "sdsa"}, {"sender": "xcsadc", "created_at": "2019-04-12T12:50:57.283147+00:00", "text": "‍❤️‍‍"}, {"sender": "xcsadc", "created_at": "2019-04-12T12:39:35.248517+00:00", "text": "czx"}, {"sender": "xcsadc", "created_at": "2019-04-12T12:39:34.352752+00:00", "text": "dsad"}, {"sender": "xcsadc", "created_at": "2019-04-12T12:39:30.889023+00:00", "media": "https://scontent-lax3-1.cdninstagram.com/vp/e985406d6eac06bb11c2d6052c1821a2/5D508106/t51.2885-15/e15/s640x640/56218099_577906226037731_8663356006073884002_n.jpg?_nc_ht=scontent-lax3-1.cdninstagram.com&ig_cache_key=Mjg2ODYwMjk0MjA1ODQxNzYzNjM1OTI1ODMwMjYzMTExNjg%3D.2"}, {"sender": "xcsadc", "created_at": "2019-04-12T12:38:54.823472+00:00", "text": "hi hi hi"}]}]

预期结果

https://scontent-lax3-1.cdninstagram.com/vp/edddf95178aca7bf75930ab8698ee45b/5D45203B/t51.2885-15/fr/e15/s1080x1080/55823673_114448266206459_7321604432125975069_n.jpg?_nc_ht=scontent-lax3-1.cdninstagram.com&ig_cache_key=Mjg2ODYwNDMxNzU3OTI1MTE1NTAxNjQ1NTk5MDkwOTMzNzY%3D.2

https://scontent-lax3-1.cdninstagram.com/vp/e985406d6eac06bb11c2d6052c1821a2/5D508106/t51.2885-15/e15/s640x640/56218099_577906226037731_8663356006073884002_n.jpg?_nc_ht=scontent-lax3-1.cdninstagram.com&ig_cache_key=Mjg2ODYwMjk0MjA1ODQxNzYzNjM1OTI1ODMwMjYzMTExNjg%3D.2

更新的

{"sender": "ncccy", "created_at": "2019-01-28T17:09:29.216184+00:00", "media": "https://scontent-lax3-1.cdninstagram.com/vp/57c43d748xcasc1abf58c890c5a6df042/5D199AE8/t51.2885-15/e15/p480x480/49913269_2181952555454636_8892094125900591548_n.jpg?_nc_ht=scontent-lax3-1.cdninstagram.com&ig_cache_key=Mjg1NjgsdasdAwNjgxNTk1OTY0OTIwMTA1NTMzNDQ%3D.2"},
  • Ctrl+H
  • 查找内容:(?:^|\G).*?"media": "(https://[^"]+)(?:(?!https:).)*
  • 替换为:\n
  • 选中环绕
  • 检查正则表达式
  • 取消勾选. matches newline
  • 全部替换

解释:

(?:^|\G)            # beginning of line OR restart from last match position
.*?                 # 0 or more any character but newline, not greedy
"media": "          # literally
(                   # start group 1
  https://[^"]+     # https:// fllowed by 1 or more not double quote, the url
)                   # end group 1
(?:(?!https:).)*    # Tempered greedy token, make sure we haven't "https" after

替换:

         # content of group 1, the URL

给定示例的结果:

https://scontent-lax3-1.cdninstagram.com/vp/edddf95178aca7bf75930ab8698ee45b/5D45203B/t51.2885-15/fr/e15/s1080x1080/55823673_114448266206459_7321604432125975069_n.jpg?_nc_ht=scontent-lax3-1.cdninstagram.com&ig_cache_key=Mjg2ODYwNDMxNzU3OTI1MTE1NTAxNjQ1NTk5MDkwOTMzNzY%3D.2
https://scontent-lax3-1.cdninstagram.com/vp/e985406d6eac06bb11c2d6052c1821a2/5D508106/t51.2885-15/e15/s640x640/56218099_577906226037731_8663356006073884002_n.jpg?_nc_ht=scontent-lax3-1.cdninstagram.com&ig_cache_key=Mjg2ODYwMjk0MjA1ODQxNzYzNjM1OTI1ODMwMjYzMTExNjg%3D.2

屏幕截图:

试试这个:

首先,我们将删除所有不属于有效 url 的字符以及引号和空格。这将删除在某些情况下似乎会导致 trouble 在记事本 ++ 上使用 boost 正则表达式的表情符号。

我们的第一个替补将是:

搜索:[^a-zA-Z0-9_\-.~:\/?#\[\]@!$&'()*+,;=%"\s]

替换为:(leave empty)

全部替换

(notepad++ 的未来版本可能不需要上一步)

清理后,我们进行以下替换:

搜索:(?i)(?:(?:(?!https?:).(?!https?:))*?"sender"\s*+:\s*+"([^"]*)"|\G)(?:.(?!"sender"\s*+:\s*+))*?(https?:.*?(?=[^a-zA-Z0-9_\-.~:\/?#\[\]@!$&'()*+,;=%]|https?:))|.*

替换:(?{1}\n\n\t:(?{2}\t)

全部替换

即使 "text" 属性内部有多个 url,这也应该有效。 url 将由制表符分隔。

所以在将前面的过程应用到这个数据之后:

[{"participants": ["minanageh379", "xcsadc"], "conversation": [{"sender": "minanageh379", "created_at": "2019-04-12T12:51:56.560361+00:00", "media": "https://scontent-lax3-1.cdninstagram.com/vp/edddf95178aca7bf75930ab8698ee45b/5D45203B/t51.2885-15/fr/e15/s1080x1080/55823673_114448266206459_7321604432125975069_n.jpg?_nc_ht=scontent-lax3-1.cdninstagram.com&ig_cache_key=Mjg2ODYwNDMxNzU3OTI1MTE1NTAxNjQ1NTk5MDkwOTMzNzY%3D.2   http://foo.barhttps://bar.foo"}, {"sender": "minanageh379", "created_at": "2019-04-12T12:51:51.923138+00:00", "text": "sd"}, {"sender": "minanageh379", "created_at": "2019-04-12T12:51:41.689524+00:00", "text": "sdsa"}, {"sender": "xcsadc", "created_at": "2019-04-12T12:50:57.283147+00:00", "text": "‍❤️‍‍"}, {"sender": "xcsadc", "created_at": "2019-04-12T12:39:35.248517+00:00", "text": "czx"}, {"sender": "xcsadc", "created_at": "2019-04-12T12:39:34.352752+00:00", "text": "dsad"}, {"sender": "xcsadc", "created_at": "2019-04-12T12:39:30.889023+00:00", "media": "https://scontent-lax3-1.cdninstagram.com/vp/e985406d6eac06bb11c2d6052c1821a2/5D508106/t51.2885-15/e15/s640x640/56218099_577906226037731_8663356006073884002_n.jpg?_nc_ht=scontent-lax3-1.cdninstagram.com&ig_cache_key=Mjg2ODYwMjk0MjA1ODQxNzYzNjM1OTI1ODMwMjYzMTExNjg%3D.2"}, {"sender": "xcsadc", "created_at": "2019-04-12T12:38:54.823472+00:00", "text": "hi hi hi"}, {"sender": "no_media_no_text", "created_at": "2019-04-12T12:38:54.823472+00:00"}, {"sender": "url_inside_text", "created_at": "2019-04-12T12:38:54.823472+00:00", "text": "Hi! {check} this url: \"http://foo.bar\" another url: https://new.url.com/ yet another one: https://google.com/"}, {"sender": "ncccy", "created_at": "2019-01-28T17:09:29.216184+00:00", "media": "https://scontent-lax3-1.cdninstagram.com/vp/57c43d748xcasc1abf58c890c5a6df042/5D199AE8/t51.2885-15/e15/p480x480/49913269_2181952555454636_8892094125900591548_n.jpg?_nc_ht=scontent-lax3-1.cdninstagram.com&ig_cache_key=Mjg1NjgsdasdAwNjgxNTk1OTY0OTIwMTA1NTMzNDQ%3D.2"}, {"sender": "ny", "created_at": "2017-10-22T20:49:50.042588+00:00", "media": "https://scontent-lax3-1.cdninstagram.com/vp/19d94ea45c2102a0f7c97838ef546b93/5D14B3C3/t51.2885-15/e15/22708873_149637425772501_5029503881546039296_n.jpg?_nc_ht=scontent-lax3-1.cdninstagram.com&ig_cache_key=Mjc4MzA3MDIyMTI3NDE3Njc3NTQxNTM1NTI2MjQyMjIyMDg%3D.2"}, {"sender": "xcsadc", "created_at": "2019-04-12T12:39:35.248517+00:00", "text": "czx"}]}]

我们得到:

minanageh379    https://scontent-lax3-1.cdninstagram.com/vp/edddf95178aca7bf75930ab8698ee45b/5D45203B/t51.2885-15/fr/e15/s1080x1080/55823673_114448266206459_7321604432125975069_n.jpg?_nc_ht=scontent-lax3-1.cdninstagram.com&ig_cache_key=Mjg2ODYwNDMxNzU3OTI1MTE1NTAxNjQ1NTk5MDkwOTMzNzY%3D.2    http://foo.bar  https://bar.foo

xcsadc  https://scontent-lax3-1.cdninstagram.com/vp/e985406d6eac06bb11c2d6052c1821a2/5D508106/t51.2885-15/e15/s640x640/56218099_577906226037731_8663356006073884002_n.jpg?_nc_ht=scontent-lax3-1.cdninstagram.com&ig_cache_key=Mjg2ODYwMjk0MjA1ODQxNzYzNjM1OTI1ODMwMjYzMTExNjg%3D.2

url_inside_text http://foo.bar  https://new.url.com/    https://google.com/

ncccy   https://scontent-lax3-1.cdninstagram.com/vp/57c43d748xcasc1abf58c890c5a6df042/5D199AE8/t51.2885-15/e15/p480x480/49913269_2181952555454636_8892094125900591548_n.jpg?_nc_ht=scontent-lax3-1.cdninstagram.com&ig_cache_key=Mjg1NjgsdasdAwNjgxNTk1OTY0OTIwMTA1NTMzNDQ%3D.2

ny  https://scontent-lax3-1.cdninstagram.com/vp/19d94ea45c2102a0f7c97838ef546b93/5D14B3C3/t51.2885-15/e15/22708873_149637425772501_5029503881546039296_n.jpg?_nc_ht=scontent-lax3-1.cdninstagram.com&ig_cache_key=Mjc4MzA3MDIyMTI3NDE3Njc3NTQxNTM1NTI2MjQyMjIyMDg%3D.2

如果在原始输入上(在相同或不同的属性上)重复,您可能会得到重复的 url。

处理后,您可以使用此正则表达式删除重复项:

搜索:(?i)\t(https?:\S++)(?=[^\n]+)

替换为:(nothing)

全部替换

要仅从文本文件中提取 links,请使用以下内容执行正则表达式“全部替换”:

查找内容:

.*?(https?:[^"]+)(?(?!.*?https?:).*)

替换为:

\n\n

Demo 1

请注意,您需要检查 Wrap around 以防插入点不在文本的开头。

解释:

.*?(https?:[^"]+)(?(?!.*?https?:).*)
|_||____________||_________________|
 |    ____|               |
 |   |    ________________|
 |   |   |
 |   |  [3] If there are no more following links, grab and discard the rest of the text
 |  [2] Store the link in  (starting with http and ending just before the first following")
[1] Grab and discard everything up 'til the first link (i.e. starting with http: or https:)

使用“全部替换”时,搜索和替换会自动继续,直到正则表达式无法匹配为止,从数据匹配的最后一点开始直到,在这种情况下,该点将在末尾的双引号之前如果还有更多 link,则当前 link 的序号,否则为文本的末尾。



要同时提取发件人,请使用以下内容:

查找内容:

.*?\{(?:([^"]*)"){4}[^{}]*?(https?:[^"]+)(?(?!.*?https?:).*)

替换为:

 \n\n

Demo 2

解释:

明天来


另一个执行相同操作的正则表达式,但可能更清楚一点:

.*?"sender": "([^"]*)[^}]*?(https?:[^"]+)(?(?!.*?https?:).*)

Demo 3

解释:

.*?"sender": "([^"]*)[^}]*?(https?:[^"]+)(?(?!.*?https?:).*)
|_||_________||_____||____||____________||_________________|
 |   ___|  ______|  ___|  _______|  _____________|
 |  |   __|  ______|  ___|  _______|
 |  |  |   _|   _____|  ___|
 |  |  |  |   _|  _____|
 |  |  |  |  |   |
 |  |  |  |  |  [6] If there are no more following links, grab and discard the rest of the text
 |  |  |  | [5] Store the link in  (starting with http and ending just before the first following")
 |  |  | [4] Grab and discard everything within the current set of braces up 'til the link
 |  | [3] Store the sender name in  
 | [2] Grab and discard "sender": " (i.e. up to the opening quote of the sender name)
[1] Grab and discard everything up 'til the first "sender" key which has an associated link

步骤 [1] 首先从文本的开头开始抓取所有内容直到第一个发件人密钥,然后通过 [2] 抓取密钥,在 [3] 中抓取发件人姓名,然后抓取所有内容直到关联的 link(如果它存在于 [4] 中)。如果没有关联的 link,[5] 失败,并且正则表达式回溯到步骤 [1],该步骤继续抓取第一个发件人密钥之后的所有内容,直到第二个发件人密钥。重复此循环,直到找到具有关联 link.

的发件人密钥

此时,步骤 [5] 成功,然后步骤 [6] 要么抓取文本的其余部分,要么什么都不抓取。

最后,所有抓取的文本都被 \n\n 替换,即发件人姓名后跟 space、link 和两个换行符。

第一个 "replace" 到此结束。由于选择了“全部替换”,整个过程再次开始,但文本指针要么位于先前找到的 link 末尾的双引号处,要么位于文本的末尾而不是开头。

虽然其他答案完全符合您的需要,但需要注意的一件事是您提供的字符串是有效的 JSON 字符串。您可以验证它是否有效 JSON here

如果您在程序中处理此字符串,您可能需要考虑使用适合您的语言的 JSON 解析器。 Here's the one for Python

另一种选择是解析 JSON 数据。

您可以使用 javascript 执行此操作。

以下代码片段应该可以解析您的数据。它甚至可以在同一 text 消息中使用多个网址:

yourJSON
[0].conversation
.filter(x => x.media !== undefined || x.text !== undefined && /https?:/i.test(x.text))
.map(x => {
    const tmp = x.text + ' ' + x.media;
    const urls = tmp.match(/https?:[\w\-.~:\/?#\[\]@!$&'()*+,;=%]*/g);
    return x.sender + ":\n" + urls.join("\n");
})
.join("\n\n");

您可以将 javascript(用您的数据更改 yourJSON)粘贴到具有某些 javascript 控制台的浏览器中,例如 Firefox 或 Chrome。在 Firefox 中,您可以使用 (Control + Shift + K) 启动控制台,在 Chrome 中使用 (Control + Shift + I,然后单击 'console')

作为替代方案,您可以使用此 jsfiddle

编辑 javascript 方块以使用您的数据,然后按 "Run" 按钮。