如何在 Flutter 中从 HTML 元素访问数据（抓取网站）？

Question

我目前在抓取网站数据（特别是 mooncalc.org）时遇到了两难，我相信，这是一个非常薄弱的信念，即我已经获得了数据并且正在评估正确的部分。

资源： https://itnext.io/write-your-first-web-scraper-in-dart-243c7bb4d05 https://www.mooncalc.org/#/48.8583,-71.3292,10/2019.12.25/20:10/1/0 通过开发者工具（我特别想获得月球年龄，但我希望这个问题得到解决而不是具体细节）

（其他：我发现 mooncalc 声称它有一个 API 非常烦人，但 link 只是指向一个带有 [=80= 的网站] 所以，如果我可以从那里的网站上免费获取数据，就不要打扰他们。)

我一直在尝试获取 HTML 并使用 HTPL 和 HTML flutter 依赖项对其进行解析。

https://pub.dev/packages/http https://pub.dev/packages/html#-changelog-tab-

  int process = 0;


LocationData _location;
  double _lunarDay;
  DateTime _nextEkadashi;
  @override
  void initState() {
    super.initState();
    if (process == 0) {
      getLocation();
      // initiate();
    }
  }

  Future initiate() async {
    var client = http.Client();
    http.Response response = await client.get(
        'https://mooncalc.org/#/${_location.latitude},${_location.longitude},null/null/null/null/null');
    ok.Document document = parse(response.body);
List<ok.Element> maybe1 = document.getElementsByClassName("moontext alter");
print(maybe1[0].text);
  }

请想象一下位置数据，现在这个 returns ...，如果没有印刷中的 .text，它 returns HTML span>（或与此编辑器类似的东西实际上是一个 HTML 跨度），我如何通过漂亮的 flutter 或命令行读取该数据。另外，列表中只有一项

另外我更喜欢flutter 运行 F5。

import 'package:flutter/material.dart';
import 'package:flutter/services.dart';
import 'package:html/dom.dart' as ok;
import 'dart:convert';

import 'package:location/location.dart';
import 'package:http/http.dart' as http;
import 'package:intl/intl.dart';
import 'package:html/parser.dart';

这些是我的导入，有多个冲突所以我不得不重命名它们。

P运行its-iMac:Ekadashi p运行itshah$ 扑动分析正在分析 Ekadashi...

没有发现问题！（运行在 332.3 秒内） P运行its-iMac:Ekadashi p运行itshah$ flutter doctor -v [✓] Flutter (Channel master, v1.13.6-pre.22, on Mac OS X 10.13.6 17G9016，语言环境 en-US） • Flutter 版本 1.13.6-pre.22 在 /Users/pranitshah/Developer/flutter • 框架修订版 d874596e38（2 天前），2019-12-23 16:16:43-0800 • 引擎版本 33813929e3 • Dart 版本 2.8.0（构建 2.8.0-dev.0.0 886615d0f9）

[✓] Android 工具链 - 为 Android 设备开发（Android SDK 版本 29.0.2) • Android SDK /Users/pranitshah/Library/Android/sdk • Android NDK 位置未配置（可选；对本机分析支持） • 平台 android-29，构建工具 29.0.2 • Java 二进制位于：/Applications/Android 工作室。app/Contents/jre/jdk/Contents/Home/bin/java • Java 版本 OpenJDK 运行时环境（构建 1.8.0_152-release-1343-b01) • 已接受所有 Android 个许可。

[✓] Xcode - 为 iOS 和 macOS 开发 (Xcode 10.2.1) • Xcode 在 /Applications/Xcode10.2.1.app/Contents/Developer • Xcode 10.2.1，内部版本 10E1001 • CocoaPods 版本 1.7.5

[✓] Chrome - 为网络开发 • Chrome 在 /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio（版本 3.4） • Android 工作室位于 /Applications/Android 工作室。app/Contents • Flutter 插件版本 38.2.1 • Dart 插件版本 183.6270 • Java 版本 OpenJDK 运行时环境（构建 1.8.0_152-release-1343-b01)

[✓] VS 代码（版本 1.41.1） • /Applications/Visual Studio Code 中的 VS Code。app/Contents • Flutter 扩展版本 3.7.1

[✓] 连接的设备（3 个可用） • 三星 SM J120A • 42007cb0e416a3bd • android-arm • Android6.0.1（API23） • Chrome • chrome • web-javascript • Google Chrome79.0.3945.88 • 网络服务器 • 网络服务器 • 网络-javascript • 颤动工具

• 未发现问题！

Answer 1

您可以将网站转换为 JSON，然后将 JSON 加载到模型中。
要将网站转换为 JSON，您需要 jsonml converter 之类的东西。例如：

我尝试将 HTML 转换为 JSON 以获得 mooncalc.org ，我得到了这个输出。

    class HtmltoModel {
    Head head;
    List<Null> headers;
    List<Null> jsonLd;
    List<Forms> forms;
    List<Tables> tables;
    List<Null> iframes;
    List<Null> embeds;
    List<Imgs> imgs;
    List<Links> links;
    String content;
    Null datetime;

    HtmltoModel({this.head, this.headers, this.jsonLd, this.forms, this.tables, this.iframes, this.embeds, this.imgs, this.links, this.content, this.datetime});

    HtmltoModel.fromJson(Map<String, dynamic> json) {
        head = json['head'] != null ? new Head.fromJson(json['head']) : null;
        if (json['headers'] != null) {
            headers = new List<Null>();
            json['headers'].forEach((v) { headers.add(new Null.fromJson(v)); });
        }
        if (json['jsonLd'] != null) {
            jsonLd = new List<Null>();
            json['jsonLd'].forEach((v) { jsonLd.add(new Null.fromJson(v)); });
        }
        if (json['forms'] != null) {
            forms = new List<Forms>();
            json['forms'].forEach((v) { forms.add(new Forms.fromJson(v)); });
        }
        if (json['tables'] != null) {
            tables = new List<Tables>();
            json['tables'].forEach((v) { tables.add(new Tables.fromJson(v)); });
        }
        if (json['iframes'] != null) {
            iframes = new List<Null>();
            json['iframes'].forEach((v) { iframes.add(new Null.fromJson(v)); });
        }
        if (json['embeds'] != null) {
            embeds = new List<Null>();
            json['embeds'].forEach((v) { embeds.add(new Null.fromJson(v)); });
        }
        if (json['imgs'] != null) {
            imgs = new List<Imgs>();
            json['imgs'].forEach((v) { imgs.add(new Imgs.fromJson(v)); });
        }
        if (json['links'] != null) {
            links = new List<Links>();
            json['links'].forEach((v) { links.add(new Links.fromJson(v)); });
        }
        content = json['content'];
        datetime = json['datetime'];
    }

    Map<String, dynamic> toJson() {
        final Map<String, dynamic> data = new Map<String, dynamic>();
        if (this.head != null) {
      data['head'] = this.head.toJson();
    }
        if (this.headers != null) {
      data['headers'] = this.headers.map((v) => v.toJson()).toList();
    }
        if (this.jsonLd != null) {
      data['jsonLd'] = this.jsonLd.map((v) => v.toJson()).toList();
    }
        if (this.forms != null) {
      data['forms'] = this.forms.map((v) => v.toJson()).toList();
    }
        if (this.tables != null) {
      data['tables'] = this.tables.map((v) => v.toJson()).toList();
    }
        if (this.iframes != null) {
      data['iframes'] = this.iframes.map((v) => v.toJson()).toList();
    }
        if (this.embeds != null) {
      data['embeds'] = this.embeds.map((v) => v.toJson()).toList();
    }
        if (this.imgs != null) {
      data['imgs'] = this.imgs.map((v) => v.toJson()).toList();
    }
        if (this.links != null) {
      data['links'] = this.links.map((v) => v.toJson()).toList();
    }
        data['content'] = this.content;
        data['datetime'] = this.datetime;
        return data;
    }
}

class Head {
    String title;
    List<Meta> meta;
    List<Link> link;
    List<Script> script;

    Head({this.title, this.meta, this.link, this.script});

    Head.fromJson(Map<String, dynamic> json) {
        title = json['title'];
        if (json['meta'] != null) {
            meta = new List<Meta>();
            json['meta'].forEach((v) { meta.add(new Meta.fromJson(v)); });
        }
        if (json['link'] != null) {
            link = new List<Link>();
            json['link'].forEach((v) { link.add(new Link.fromJson(v)); });
        }
        if (json['script'] != null) {
            script = new List<Script>();
            json['script'].forEach((v) { script.add(new Script.fromJson(v)); });
        }
    }

    Map<String, dynamic> toJson() {
        final Map<String, dynamic> data = new Map<String, dynamic>();
        data['title'] = this.title;
        if (this.meta != null) {
      data['meta'] = this.meta.map((v) => v.toJson()).toList();
    }
        if (this.link != null) {
      data['link'] = this.link.map((v) => v.toJson()).toList();
    }
        if (this.script != null) {
      data['script'] = this.script.map((v) => v.toJson()).toList();
    }
        return data;
    }
}

class Meta {
    String charset;
    String name;
    String content;
    String property;

    Meta({this.charset, this.name, this.content, this.property});

    Meta.fromJson(Map<String, dynamic> json) {
        charset = json['charset'];
        name = json['name'];
        content = json['content'];
        property = json['property'];
    }

    Map<String, dynamic> toJson() {
        final Map<String, dynamic> data = new Map<String, dynamic>();
        data['charset'] = this.charset;
        data['name'] = this.name;
        data['content'] = this.content;
        data['property'] = this.property;
        return data;
    }
}

class Link {
    String rel;
    String href;
    String sizes;
    String type;
    String media;

    Link({this.rel, this.href, this.sizes, this.type, this.media});

    Link.fromJson(Map<String, dynamic> json) {
        rel = json['rel'];
        href = json['href'];
        sizes = json['sizes'];
        type = json['type'];
        media = json['media'];
    }

    Map<String, dynamic> toJson() {
        final Map<String, dynamic> data = new Map<String, dynamic>();
        data['rel'] = this.rel;
        data['href'] = this.href;
        data['sizes'] = this.sizes;
        data['type'] = this.type;
        data['media'] = this.media;
        return data;
    }
}

class Script {
    String src;
    String type;
    String async;
    String id;
    String dataBg;
    String dataFg;
    String dataLink;
    String dataLinkmsg;
    String dataLinktarget;
    String dataEffect;
    String dataCloseText;
    String dataMoreinfo;
    String dataCookie;
    String dataTextAlign;
    String dataFontSize;
    String dataDivlinkbg;
    String dataDivlink;
    String dataZindex;
    String dataMaxage;
    String dataOpacity;

    Script({this.src, this.type, this.async, this.id, this.dataBg, this.dataFg, this.dataLink, this.dataLinkmsg, this.dataLinktarget, this.dataEffect, this.dataCloseText, this.dataMoreinfo, this.dataCookie, this.dataTextAlign, this.dataFontSize, this.dataDivlinkbg, this.dataDivlink, this.dataZindex, this.dataMaxage, this.dataOpacity});

    Script.fromJson(Map<String, dynamic> json) {
        src = json['src'];
        type = json['type'];
        async = json['async'];
        id = json['id'];
        dataBg = json['data-bg'];
        dataFg = json['data-fg'];
        dataLink = json['data-link'];
        dataLinkmsg = json['data-linkmsg'];
        dataLinktarget = json['data-linktarget'];
        dataEffect = json['data-effect'];
        dataCloseText = json['data-close-text'];
        dataMoreinfo = json['data-moreinfo'];
        dataCookie = json['data-cookie'];
        dataTextAlign = json['data-text-align'];
        dataFontSize = json['data-font-size'];
        dataDivlinkbg = json['data-divlinkbg'];
        dataDivlink = json['data-divlink'];
        dataZindex = json['data-zindex'];
        dataMaxage = json['data-maxage'];
        dataOpacity = json['data-opacity'];
    }

    Map<String, dynamic> toJson() {
        final Map<String, dynamic> data = new Map<String, dynamic>();
        data['src'] = this.src;
        data['type'] = this.type;
        data['async'] = this.async;
        data['id'] = this.id;
        data['data-bg'] = this.dataBg;
        data['data-fg'] = this.dataFg;
        data['data-link'] = this.dataLink;
        data['data-linkmsg'] = this.dataLinkmsg;
        data['data-linktarget'] = this.dataLinktarget;
        data['data-effect'] = this.dataEffect;
        data['data-close-text'] = this.dataCloseText;
        data['data-moreinfo'] = this.dataMoreinfo;
        data['data-cookie'] = this.dataCookie;
        data['data-text-align'] = this.dataTextAlign;
        data['data-font-size'] = this.dataFontSize;
        data['data-divlinkbg'] = this.dataDivlinkbg;
        data['data-divlink'] = this.dataDivlink;
        data['data-zindex'] = this.dataZindex;
        data['data-maxage'] = this.dataMaxage;
        data['data-opacity'] = this.dataOpacity;
        return data;
    }
}

class Forms {
    Attr attr;
    List<Fields> fields;

    Forms({this.attr, this.fields});

    Forms.fromJson(Map<String, dynamic> json) {
        attr = json['attr'] != null ? new Attr.fromJson(json['attr']) : null;
        if (json['fields'] != null) {
            fields = new List<Fields>();
            json['fields'].forEach((v) { fields.add(new Fields.fromJson(v)); });
        }
    }

    Map<String, dynamic> toJson() {
        final Map<String, dynamic> data = new Map<String, dynamic>();
        if (this.attr != null) {
      data['attr'] = this.attr.toJson();
    }
        if (this.fields != null) {
      data['fields'] = this.fields.map((v) => v.toJson()).toList();
    }
        return data;
    }
}

class Attr {
    String id;

    Attr({this.id});

    Attr.fromJson(Map<String, dynamic> json) {
        id = json['id'];
    }

    Map<String, dynamic> toJson() {
        final Map<String, dynamic> data = new Map<String, dynamic>();
        data['id'] = this.id;
        return data;
    }
}

class Fields {
    String node;
    String name;
    Attr attr;

    Fields({this.node, this.name, this.attr});

    Fields.fromJson(Map<String, dynamic> json) {
        node = json['node'];
        name = json['name'];
        attr = json['attr'] != null ? new Attr.fromJson(json['attr']) : null;
    }

    Map<String, dynamic> toJson() {
        final Map<String, dynamic> data = new Map<String, dynamic>();
        data['node'] = this.node;
        data['name'] = this.name;
        if (this.attr != null) {
      data['attr'] = this.attr.toJson();
    }
        return data;
    }
}

class Attr {
    String type;
    String id;
    String name;
    String value;
    String size;
    String maxlength;
    String class;
    String title;

    Attr({this.type, this.id, this.name, this.value, this.size, this.maxlength, this.class, this.title});

    Attr.fromJson(Map<String, dynamic> json) {
        type = json['type'];
        id = json['id'];
        name = json['name'];
        value = json['value'];
        size = json['size'];
        maxlength = json['maxlength'];
        class = json['class'];
        title = json['title'];
    }

    Map<String, dynamic> toJson() {
        final Map<String, dynamic> data = new Map<String, dynamic>();
        data['type'] = this.type;
        data['id'] = this.id;
        data['name'] = this.name;
        data['value'] = this.value;
        data['size'] = this.size;
        data['maxlength'] = this.maxlength;
        data['class'] = this.class;
        data['title'] = this.title;
        return data;
    }
}

class Tables {
    Attr attr;
    List<Rows> rows;

    Tables({this.attr, this.rows});

    Tables.fromJson(Map<String, dynamic> json) {
        attr = json['attr'] != null ? new Attr.fromJson(json['attr']) : null;
        if (json['rows'] != null) {
            rows = new List<Rows>();
            json['rows'].forEach((v) { rows.add(new Rows.fromJson(v)); });
        }
    }

    Map<String, dynamic> toJson() {
        final Map<String, dynamic> data = new Map<String, dynamic>();
        if (this.attr != null) {
      data['attr'] = this.attr.toJson();
    }
        if (this.rows != null) {
      data['rows'] = this.rows.map((v) => v.toJson()).toList();
    }
        return data;
    }
}

class Attr {
    String id;
    String width;
    String cellspacing;
    String style;

    Attr({this.id, this.width, this.cellspacing, this.style});

    Attr.fromJson(Map<String, dynamic> json) {
        id = json['id'];
        width = json['width'];
        cellspacing = json['cellspacing'];
        style = json['style'];
    }

    Map<String, dynamic> toJson() {
        final Map<String, dynamic> data = new Map<String, dynamic>();
        data['id'] = this.id;
        data['width'] = this.width;
        data['cellspacing'] = this.cellspacing;
        data['style'] = this.style;
        return data;
    }
}

class Rows {
    List<Null> attr;
    List<Cols> cols;

    Rows({this.attr, this.cols});

    Rows.fromJson(Map<String, dynamic> json) {
        if (json['attr'] != null) {
            attr = new List<Null>();
            json['attr'].forEach((v) { attr.add(new Null.fromJson(v)); });
        }
        if (json['cols'] != null) {
            cols = new List<Cols>();
            json['cols'].forEach((v) { cols.add(new Cols.fromJson(v)); });
        }
    }

    Map<String, dynamic> toJson() {
        final Map<String, dynamic> data = new Map<String, dynamic>();
        if (this.attr != null) {
      data['attr'] = this.attr.map((v) => v.toJson()).toList();
    }
        if (this.cols != null) {
      data['cols'] = this.cols.map((v) => v.toJson()).toList();
    }
        return data;
    }
}

class Cols {
    List<Null> attr;
    String nodeValue;

    Cols({this.attr, this.nodeValue});

    Cols.fromJson(Map<String, dynamic> json) {
        if (json['attr'] != null) {
            attr = new List<Null>();
            json['attr'].forEach((v) { attr.add(new Null.fromJson(v)); });
        }
        nodeValue = json['nodeValue'];
    }

    Map<String, dynamic> toJson() {
        final Map<String, dynamic> data = new Map<String, dynamic>();
        if (this.attr != null) {
      data['attr'] = this.attr.map((v) => v.toJson()).toList();
    }
        data['nodeValue'] = this.nodeValue;
        return data;
    }
}

class Imgs {
    String src;
    Attr attr;

    Imgs({this.src, this.attr});

    Imgs.fromJson(Map<String, dynamic> json) {
        src = json['src'];
        attr = json['attr'] != null ? new Attr.fromJson(json['attr']) : null;
    }

    Map<String, dynamic> toJson() {
        final Map<String, dynamic> data = new Map<String, dynamic>();
        data['src'] = this.src;
        if (this.attr != null) {
      data['attr'] = this.attr.toJson();
    }
        return data;
    }
}

class Attr {
    String src;
    String style;
    String title;
    String border;
    String align;

    Attr({this.src, this.style, this.title, this.border, this.align});

    Attr.fromJson(Map<String, dynamic> json) {
        src = json['src'];
        style = json['style'];
        title = json['title'];
        border = json['border'];
        align = json['align'];
    }

    Map<String, dynamic> toJson() {
        final Map<String, dynamic> data = new Map<String, dynamic>();
        data['src'] = this.src;
        data['style'] = this.style;
        data['title'] = this.title;
        data['border'] = this.border;
        data['align'] = this.align;
        return data;
    }
}

class Links {
    String text;
    String href;
    Attr attr;

    Links({this.text, this.href, this.attr});

    Links.fromJson(Map<String, dynamic> json) {
        text = json['text'];
        href = json['href'];
        attr = json['attr'] != null ? new Attr.fromJson(json['attr']) : null;
    }

    Map<String, dynamic> toJson() {
        final Map<String, dynamic> data = new Map<String, dynamic>();
        data['text'] = this.text;
        data['href'] = this.href;
        if (this.attr != null) {
      data['attr'] = this.attr.toJson();
    }
        return data;
    }
}

class Attr {
    String href;
    String id;
    String class;
    String title;
    String rel;
    String dataOpenimage;
    String dataClosedimage;
    String target;
    String style;

    Attr({this.href, this.id, this.class, this.title, this.rel, this.dataOpenimage, this.dataClosedimage, this.target, this.style});

    Attr.fromJson(Map<String, dynamic> json) {
        href = json['href'];
        id = json['id'];
        class = json['class'];
        title = json['title'];
        rel = json['rel'];
        dataOpenimage = json['data-openimage'];
        dataClosedimage = json['data-closedimage'];
        target = json['target'];
        style = json['style'];
    }

    Map<String, dynamic> toJson() {
        final Map<String, dynamic> data = new Map<String, dynamic>();
        data['href'] = this.href;
        data['id'] = this.id;
        data['class'] = this.class;
        data['title'] = this.title;
        data['rel'] = this.rel;
        data['data-openimage'] = this.dataOpenimage;
        data['data-closedimage'] = this.dataClosedimage;
        data['target'] = this.target;
        data['style'] = this.style;
        return data;
    }
}

如何在 Flutter 中从 HTML 元素访问数据（抓取网站）？

How to Access Data from HTML Element (Scrape a Website) in Flutter?

client

http

httprequest

html-parsing

flutter