分层数据聚合和操作
Hierarchical data aggregation and manipulation
我是 r 的新手，我正在尝试处理我的数据中的一些不一致之处。
我的问题是双重的，第一部分可能是普遍感兴趣的，它是关于如何聚合分类在具有多层次聚合的向量中的数据。第二个问题与我的编码问题更密切相关，它是关于对我的数据执行一些特定的操作。
我正在查看二十年来数百个国家/地区的出口数据。问题是我的出口数据按产品和子产品类别（数百种）分类，方式不一致，我正在努力处理这些差异。
数据大致如下：
df <- data.frame(
"Reporter" = c("USA", "USA", "USA", "USA", "USA", "USA","USA","EU", "EU","EU", "EU", "EU", "EU", "EU", "EU"),
"Partner" = c( "EU", "EU","EU","EU", "EU","EU","EU","USA", "USA", "USA","USA","USA", "USA","USA", "USA"),
"Product cat." = c("1", "1.1", "1.2","2", "2.1", "2.2","3","1", "1.1","2", "2.1", "2.2","3","3.1", "3.2"),
"Year" = c(1970, 1970, 1970, 1970, 1970, 1970,1970, 1970, 1970, 1970, 1970, 1970, 1970, 1970, 1970),
"Val" = c(200, 170, 30, 100, 50, 40, 220, 230, 180, 80, 50, 20, 170, 40, 130),
stringsAsFactors = FALSE)
产品类别 1.1（例如苹果）和 1.2（例如香蕉）是产品类别 1（例如水果）的子产品类别。产品类别 2.1 和 2.2 是产品类别的子类别。 2等等。
我的最终目标如下：
首先，子产品类别的 "value" 应始终等于较高的产品类别值。这是美国出口到欧盟的情况，产品类别 1.1 (val=170) 和 1.2 (val=30) 聚合到产品类别 1 (val=200) 的级别。然而，通常情况并非如此。
例如，在美国出口到欧盟的情况下，产品类别 2.1 (val=50) 和 2.2 (val=40) 的值小于产品类别 2 (val=100)。
为了处理这个问题，我需要创建一个新的子产品猫。理想情况下，这将（自动）将产品猫名称的开头与 K 组合（因此 2.K）。这应该给出产品类别 2 及其子产品类别 2.1 和 2.2 之间的差异值 (2.K= 100-(50+40) = 10)。
此外，我想对缺少其中一个子产品类别数据的情况应用相同的方法。一个例子是从欧盟到美国的出口，其中只有产品类别 1 和子产品类别 1.1 的值，没有关于类别 1.2 的信息。理想情况下，我会创建一个新产品 cat (1.K)，其值为产品 cat 1 (val=230) 及其子产品 cat 1.1(val = 180) 之间的差值。因此，1.k 的值将是 230-180 = 50.
第二个问题是，在某些情况下，我没有子产品类别的数据，但只有聚合级别的数据。与美国出口到欧盟产品类别 3（没有子类别）的情况一样。我想创建一个新的子产品 cat a new，它将产品 cat 的开头与 M（因此 2.M）结合起来，并合并子类别级别中未报告的产品类别级别的值。因此，例如，在美国出口到欧盟产品类别 3 (220) 的情况下，3.M = 220.
如前所述，我认为有两个步骤可以解决我的编码问题。
第一个是关于如何聚合分层数据（请注意，在我的实际数据中，我有 三个 ，而不是两个，子产品级别（例如 1 食品，1.1 水果，1.1。 1 个苹果）。理想情况下，我宁愿避免创建新列，因为我的数据集涉及数百个产品类别。
第二部分是关于执行上述具体操作：1）创建一个具有父节点和子节点之间差异的新类别，2）创建虚拟子节点。
我真的很感谢任何能帮助我的人，因为这是我论文发展的关键。
我知道这是一个复杂的问题，但也欢迎部分回答。
在此先感谢大家的帮助
==============
非常感谢 LAP 的帮助，
这是我在应用函数后遇到的真实数据问题
split2 <- lapply(split1, function(x){
y <- rbind.data.frame(x, x[1,])
y[nrow(y), "Product.cat."] <- paste0(y[nrow(y), "Prodcat2"], "k")
y[nrow(y), "Val"] <- x[1, "Val"] - sum(x[2:nrow(x), "Val"])
return(y)
})
和函数
split3 <- do.call(rbind, split2)
这里是两个splits的head的dput
>dput(Headsplit2)
list(`Algeria.United Arab Emirates.05` = structure(list(Reporter =
c("Algeria",
"Algeria", "Algeria", "Algeria"), Partner = c("United Arab Emirates",
"United Arab Emirates", "United Arab Emirates", "United Arab
Emirates"
), Year = c(2001L, 2001L, 2001L, 2001L), Product.cat. = c("05",
"052", "054", "05k"), `Commodity Description` = c("Fruit and
vegetables",
"Dried fruit including artificially dehydrated", "Vegetables, roots &
tubers, fresh or dried",
"Fruit and vegetables"), `Trade Value` =
structure(c(7.61814641291993e-319,
7.4539189922423e-319, 1.64178014113046e-320, 7.61814641291993e-319
), class = "integer64"), Prodcat1 = c("0", "0", "0", "0"), Prodcat2 =
c("05",
"05", "05", "05")), row.names = c(NA, -4L), vars = c("Reporter",
"Partner", "Prodcat2", "Year"), drop = TRUE, indices = list(0:2),
group_sizes = 3L, biggest_group_size = 3L, labels = structure(list(
Reporter = "Algeria", Partner = "United Arab Emirates", Prodcat2 =
"05",
Year = 2001L), row.names = c(NA, -1L), class = "data.frame", vars =
c("Reporter",
"Partner", "Prodcat2", "Year"), drop = TRUE), class = c("grouped_df",
"tbl_df", "tbl", "data.frame")), `Algeria.United Kingdom.05` =
structure(list(
Reporter = c("Algeria", "Algeria", "Algeria", "Algeria"),
Partner = c("United Kingdom", "United Kingdom", "United Kingdom",
"United Kingdom"), Year = c(2001L, 2001L, 2001L, 2001L),
Product.cat. = c("05", "053", "054", "05k"), `Commodity Description`
= c("Fruit and vegetables",
"Fruit,preserved and fruit preparations", "Vegetables, roots &
tubers, fresh or dried",
"Fruit and vegetables"), `Trade Value` =
structure(c(6.99399328252869e-320,
3.16547859290487e-320, 3.82802062397798e-320, 6.99399328252869e-320
), class = "integer64"), Prodcat1 = c("0", "0", "0", "0"),
Prodcat2 = c("05", "05", "05", "05")), row.names = c(NA,
-4L), vars = c("Reporter", "Partner", "Prodcat2", "Year"), drop =
TRUE, indices = list(
0:2), group_sizes = 3L, biggest_group_size = 3L, labels =
structure(list(
Reporter = "Algeria", Partner = "United Kingdom", Prodcat2 = "05",
Year = 2001L), row.names = c(NA, -1L), class = "data.frame", vars =
c("Reporter",
"Partner", "Prodcat2", "Year"), drop = TRUE), class = c("grouped_df",
"tbl_df", "tbl", "data.frame")), Hungary.Austria.26 = structure(list(
Reporter = c("Hungary", "Hungary", "Hungary", "Hungary",
"Hungary", "Hungary", "Hungary", "Hungary", "Hungary"), Partner =
c("Austria",
"Austria", "Austria", "Austria", "Austria", "Austria", "Austria",
"Austria", "Austria"), Year = c(2000L, 2001L, 2000L, 2000L,
2001L, 2000L, 2000L, 2001L, 2000L), Product.cat. = c("26",
"26", "263", "265", "265", "266", "267", "267", "26k"), `Commodity
Description` = c("Textile fibres, not manufactured, and waste",
"Textile fibres, not manufactured, and waste", "Cotton",
"Vegetable fibres,except cotton and jute", "Vegetable fibres,except
cotton and jute",
"Synthetic and regenerated artificial fibres", "Waste materials from
textile fabrics, incl.rags",
"Waste materials from textile fabrics, incl.rags", "Textile fibres,
not manufactured, and waste"
), `Trade Value` = structure(c(7.3714594359514e-318,
9.95542276370112e-318,
4.94065645841247e-320, 2.96439387504748e-320, 6.91691904177745e-320,
2.32210853545386e-319, 6.33886223614319e-318, 9.60957681161225e-318,
7.3714594359514e-318), class = "integer64"), Prodcat1 = c("2",
"2", "2", "2", "2", "2", "2", "2", "2"), Prodcat2 = c("26",
"26", "26", "26", "26", "26", "26", "26", "26")), row.names = c(NA,
-9L), vars = c("Reporter", "Partner", "Prodcat2", "Year"), drop =
TRUE, indices = list(
c(0L, 2L, 3L, 5L, 6L), c(1L, 4L, 7L)), group_sizes = c(5L,
3L), biggest_group_size = 5L, labels = structure(list(Reporter =
c("Hungary",
"Hungary"), Partner = c("Austria", "Austria"), Prodcat2 = c("26",
"26"), Year = 2000:2001), row.names = c(NA, -2L), class =
"data.frame", vars = c("Reporter",
"Partner", "Prodcat2", "Year"), drop = TRUE), class = c("grouped_df",
"tbl_df", "tbl", "data.frame")), Hungary.Belgium.26 = structure(list(
Reporter = c("Hungary", "Hungary", "Hungary", "Hungary",
"Hungary", "Hungary", "Hungary", "Hungary", "Hungary"), Partner =
c("Belgium",
"Belgium", "Belgium", "Belgium", "Belgium", "Belgium", "Belgium",
"Belgium", "Belgium"), Year = c(2000L, 2001L, 2000L, 2001L,
2000L, 2001L, 2000L, 2001L, 2000L), Product.cat. = c("26",
"26", "265", "265", "266", "266", "267", "267", "26k"), `Commodity
Description` = c("Textile fibres, not manufactured, and waste",
"Textile fibres, not manufactured, and waste", "Vegetable
fibres,except cotton and jute",
"Vegetable fibres,except cotton and jute", "Synthetic and regenerated
artificial fibres",
"Synthetic and regenerated artificial fibres", "Waste materials from
textile fabrics, incl.rags",
"Waste materials from textile fabrics, incl.rags", "Textile fibres,
not manufactured, and waste"
), `Trade Value` = structure(c(3.41893426922143e-318,
7.98410083679454e-318,
3.95252516672997e-320, 9.73309322307256e-319, 1.67488253940183e-318,
1.665001226485e-318, 8.49792910846944e-319, 7.70742407512345e-319,
3.41893426922143e-318), class = "integer64"), Prodcat1 = c("2",
"2", "2", "2", "2", "2", "2", "2", "2"), Prodcat2 = c("26",
"26", "26", "26", "26", "26", "26", "26", "26")), row.names = c(NA,
-9L), vars = c("Reporter", "Partner", "Prodcat2", "Year"), drop =
TRUE, indices = list(
c(0L, 2L, 4L, 6L), c(1L, 3L, 5L, 7L)), group_sizes = c(4L,
4L), biggest_group_size = 4L, labels = structure(list(Reporter =
c("Hungary",
"Hungary"), Partner = c("Belgium", "Belgium"), Prodcat2 = c("26",
"26"), Year = 2000:2001), row.names = c(NA, -2L), class =
"data.frame", vars = c("Reporter",
"Partner", "Prodcat2", "Year"), drop = TRUE), class = c("grouped_df",
"tbl_df", "tbl", "data.frame")), Hungary.Bulgaria.26 =
structure(list(
Reporter = c("Hungary", "Hungary", "Hungary", "Hungary",
"Hungary", "Hungary"), Partner = c("Bulgaria", "Bulgaria",
"Bulgaria", "Bulgaria", "Bulgaria", "Bulgaria"), Year = c(2000L,
2001L, 2000L, 2001L, 2000L, 2000L), Product.cat. = c("26",
"26", "266", "266", "267", "26k"), `Commodity Description` =
c("Textile fibres, not manufactured, and waste",
"Textile fibres, not manufactured, and waste", "Synthetic and
regenerated artificial fibres",
"Synthetic and regenerated artificial fibres", "Waste materials from
textile fabrics, incl.rags",
"Textile fibres, not manufactured, and waste"), `Trade Value` =
structure(c(1.55136612794151e-318,
1.53160350210786e-319, 4.94065645841247e-321, 4.94065645841247e-321,
2.96439387504748e-320, 1.55136612794151e-318), class = "integer64"),
Prodcat1 = c("2", "2", "2", "2", "2", "2"), Prodcat2 = c("26",
"26", "26", "26", "26", "26")), row.names = c(NA, -6L), vars =
c("Reporter",
"Partner", "Prodcat2", "Year"), drop = TRUE, indices = list(c(0L,
2L, 4L), c(1L, 3L)), group_sizes = 3:2, biggest_group_size = 3L,
labels = structure(list(
Reporter = c("Hungary", "Hungary"), Partner = c("Bulgaria",
"Bulgaria"), Prodcat2 = c("26", "26"), Year = 2000:2001), row.names =
c(NA,
-2L), class = "data.frame", vars = c("Reporter", "Partner",
"Prodcat2",
"Year"), drop = TRUE), class = c("grouped_df", "tbl_df", "tbl",
"data.frame")), Hungary.Canada.26 = structure(list(Reporter =
c("Hungary",
"Hungary", "Hungary"), Partner = c("Canada", "Canada", "Canada"
), Year = c(2001L, 2001L, 2001L), Product.cat. = c("26", "265",
"26k"), `Commodity Description` = c("Textile fibres, not
manufactured, and waste",
"Vegetable fibres,except cotton and jute", "Textile fibres, not
manufactured, and waste"
), `Trade Value` = structure(c(8.89318162514244e-320,
6.4228533959362e-320,
8.89318162514244e-320), class = "integer64"), Prodcat1 = c("2",
"2", "2"), Prodcat2 = c("26", "26", "26")), row.names = c(NA,
-3L), vars = c("Reporter", "Partner", "Prodcat2", "Year"), drop =
TRUE, indices = list(
0:1), group_sizes = 2L, biggest_group_size = 2L, labels =
structure(list(
Reporter = "Hungary", Partner = "Canada", Prodcat2 = "26",
Year = 2001L), row.names = c(NA, -1L), class = "data.frame", vars =
c("Reporter",
"Partner", "Prodcat2", "Year"), drop = TRUE), class = c("grouped_df",
"tbl_df", "tbl", "data.frame")))
和拆分 3
dput(Headsplit3)
structure(list(Reporter = c("Algeria", "Algeria", "Algeria",
"Algeria", "Algeria", "Algeria"), Partner = c("United Arab Emirates",
"United Arab Emirates", "United Arab Emirates", "United Arab
Emirates",
"United Kingdom", "United Kingdom"), Year = c(2001L, 2001L, 2001L,
2001L, 2001L, 2001L), Product.cat. = c("05", "052", "054", "05k",
"05", "053"), `Commodity Description` = c("Fruit and vegetables",
"Dried fruit including artificially dehydrated", "Vegetables, roots &
tubers, fresh or dried",
"Fruit and vegetables", "Fruit and vegetables", "Fruit,preserved and
fruit preparations"
), `Trade Value` = structure(c(7.61814641291993e-319,
7.4539189922423e-319,
1.64178014113046e-320, 7.61814641291993e-319, 6.99399328252869e-320,
3.16547859290487e-320), class = "integer64"), Prodcat1 = c("0",
"0", "0", "0", "0", "0"), Prodcat2 = c("05", "05", "05", "05",
"05", "05")), row.names = c(NA, -6L), class = c("grouped_df",
"tbl_df", "tbl", "data.frame"), vars = c("Reporter", "Partner",
"Prodcat2", "Year"), drop = TRUE, indices = list(0:3, 4:5),
group_sizes = c(4L,
2L), biggest_group_size = 4L, labels = structure(list(Reporter =
c("Algeria",
"Algeria"), Partner = c("United Arab Emirates", "United Kingdom"
), Prodcat2 = c("05", "05"), Year = c(2001L, 2001L)), row.names =
c(NA,
-2L), class = "data.frame", vars = c("Reporter", "Partner",
"Prodcat2",
"Year"), drop = TRUE))
如您所见，代码能够识别出阿尔及利亚出口到阿拉伯联合酋长国的 052 和 054 没有将 05 的出口相加（差异仅为 1），并且它确实正确地创建了一个变量05k，但 05k 的交易价值是 154193（= 整个 05 的交易价值）而不是 1。你知道为什么会这样吗？
编辑：好的，我想我明白了！
数据：
df <- data.frame( "Reporter" = c("USA", "USA", "USA", "USA", "USA", "USA","USA", "USA", "USA","USA"),
"Partner" = c( "EU", "EU","EU","EU", "EU","EU","EU", "EU","EU","EU"),
"Product cat." = c("1", "11","111", "12","2", "21", "211", "212", "22", "3"),
"Val" = c(200, 170, 170, 30, 100, 50, 25, 5, 40, 220), stringsAsFactors = FALSE)
我们首先创建两个辅助变量 Prodcat1 和 Prodcat2:
# create new variable Prodcat1
df1 <- df %>% group_by(Reporter, Partner) %>% mutate(Prodcat1 = str_extract(Product.cat., "^.{1}"))
# create new variable Prodcat2 for my 2nd level product category
df1 <- df1 %>% group_by(Reporter, Partner) %>% mutate(Prodcat2 = str_extract(Product.cat., "^.{2}"))
现在我们把数据分成两部分，一部分要完成，一部分不需要第三层的操作：
# to be completed
df2 <- df1 %>%
group_by(Reporter, Partner, Prodcat2) %>%
filter(sum(Val[2:n()]) < Val[1])
# no operation on third level
df3 <- df1 %>%
group_by(Reporter, Partner, Prodcat2) %>%
filter(!sum(Val[2:n()]) < Val[1] | n() == 1)
我们将 df2 拆分为 Prodcat2，控制 Reporter 和 Partner
split1 <- split(df2, interaction(df2$Reporter, df2$Partner, df2$Prodcat2))
split1 <- split1[sapply(split1, nrow) != 0]
并在必要时添加新行：
split2 <- lapply(split1, function(x){
y <- rbind.data.frame(x, x[1,])
y[nrow(y), "Product.cat."] <- paste0(y[nrow(y), "Prodcat2"], "k")
y[nrow(y), "Val"] <- x[1, "Val"] - sum(x[2:nrow(x), "Val"])
return(y)
})
然后我们第一次把数据重新拼起来然后按照原来的Product.cat..
排序
split3 <- do.call(rbind, split2)
newdf <- do.call(rbind, list(split3, df3))
newdf <- newdf %>%
arrange(Product.cat.)
目前数据：
# A tibble: 11 x 6
# Groups: Reporter, Partner, Prodcat2 [5]
Reporter Partner Product.cat. Val Prodcat1 Prodcat2
<chr> <chr> <chr> <dbl> <chr> <chr>
1 USA EU 1 200 1 NA
2 USA EU 11 170 1 11
3 USA EU 111 170 1 11
4 USA EU 12 30 1 12
5 USA EU 2 100 2 NA
6 USA EU 21 50 2 21
7 USA EU 211 25 2 21
8 USA EU 212 5 2 21
9 USA EU 21k 20 2 21
10 USA EU 22 40 2 22
11 USA EU 3 220 3 NA
现在我们进入第二层。首先我们创建三个部分：
# part to complete
df4 <- newdf %>%
group_by(Reporter, Partner, Prodcat1) %>%
filter(nchar(Product.cat.) < 3) %>%
filter(n() == 1 | sum(Val[2:n()]) < Val[1])
# third level rows, which are not necessary here
df5 <- newdf %>%
group_by(Reporter, Partner, Prodcat1) %>%
filter(nchar(Product.cat.) == 3)
# second level part already complete
df6 <- newdf %>%
group_by(Reporter, Partner, Prodcat1) %>%
filter(nchar(Product.cat.) < 3) %>%
filter(sum(Val[2:n()]) == Val[1])
我们再次拆分数据，现在按 Prodcat1，控制 Reporter 和 Partner：
split3 <- split(df4, interaction(df4$Reporter, df4$Partner, df4$Prodcat1))
split3 <- split3[sapply(split3, nrow) != 0]
我们创建新行：
split4 <- lapply(split3, function(x){
if(nrow(x) == 1){
y <- rbind.data.frame(x, x)
y[2, "Product.cat."] <- paste0(y[2, "Prodcat1"], "m")
}else{
y <- rbind.data.frame(x, x[1,])
y[nrow(y), "Product.cat."] <- paste0(y[nrow(y), "Prodcat1"], "k")
y[nrow(y), "Val"] <- x[1, "Val"] - sum(x[2:nrow(x), "Val"])
}
return(y)
})
然后我们将它们重新组合在一起，再次排序并删除辅助变量。
split5 <- do.call(rbind, split4)
finaldf <- do.call(rbind, list(split5, df5, df6))
finaldf <- finaldf %>%
ungroup() %>%
arrange(Product.cat.) %>%
select(-c("Prodcat1", "Prodcat2"))
最终数据：
# A tibble: 13 x 4
Reporter Partner Product.cat. Val
<chr> <chr> <chr> <dbl>
1 USA EU 1 200
2 USA EU 11 170
3 USA EU 111 170
4 USA EU 12 30
5 USA EU 2 100
6 USA EU 21 50
7 USA EU 211 25
8 USA EU 212 5
9 USA EU 21k 20
10 USA EU 22 40
11 USA EU 2k 10
12 USA EU 3 220
13 USA EU 3m 220
最后，我们清除了我们需要的所有临时对象的环境
rm(df1, df2, df3, df4, df5, df6, newdf, split1, split2, split3, split4, split5)
这给我们留下了原始数据集 df 和最终完成的数据集 finaldata :)
我是 r 的新手，我正在尝试处理我的数据中的一些不一致之处。 我的问题是双重的，第一部分可能是普遍感兴趣的，它是关于如何聚合分类在具有多层次聚合的向量中的数据。第二个问题与我的编码问题更密切相关，它是关于对我的数据执行一些特定的操作。
我正在查看二十年来数百个国家/地区的出口数据。问题是我的出口数据按产品和子产品类别（数百种）分类，方式不一致，我正在努力处理这些差异。
数据大致如下：
df <- data.frame(
"Reporter" = c("USA", "USA", "USA", "USA", "USA", "USA","USA","EU", "EU","EU", "EU", "EU", "EU", "EU", "EU"),
"Partner" = c( "EU", "EU","EU","EU", "EU","EU","EU","USA", "USA", "USA","USA","USA", "USA","USA", "USA"),
"Product cat." = c("1", "1.1", "1.2","2", "2.1", "2.2","3","1", "1.1","2", "2.1", "2.2","3","3.1", "3.2"),
"Year" = c(1970, 1970, 1970, 1970, 1970, 1970,1970, 1970, 1970, 1970, 1970, 1970, 1970, 1970, 1970),
"Val" = c(200, 170, 30, 100, 50, 40, 220, 230, 180, 80, 50, 20, 170, 40, 130),
stringsAsFactors = FALSE)
产品类别 1.1（例如苹果）和 1.2（例如香蕉）是产品类别 1（例如水果）的子产品类别。产品类别 2.1 和 2.2 是产品类别的子类别。 2等等。
我的最终目标如下： 首先，子产品类别的 "value" 应始终等于较高的产品类别值。这是美国出口到欧盟的情况，产品类别 1.1 (val=170) 和 1.2 (val=30) 聚合到产品类别 1 (val=200) 的级别。然而，通常情况并非如此。 例如，在美国出口到欧盟的情况下，产品类别 2.1 (val=50) 和 2.2 (val=40) 的值小于产品类别 2 (val=100)。 为了处理这个问题，我需要创建一个新的子产品猫。理想情况下，这将（自动）将产品猫名称的开头与 K 组合（因此 2.K）。这应该给出产品类别 2 及其子产品类别 2.1 和 2.2 之间的差异值 (2.K= 100-(50+40) = 10)。 此外，我想对缺少其中一个子产品类别数据的情况应用相同的方法。一个例子是从欧盟到美国的出口，其中只有产品类别 1 和子产品类别 1.1 的值，没有关于类别 1.2 的信息。理想情况下，我会创建一个新产品 cat (1.K)，其值为产品 cat 1 (val=230) 及其子产品 cat 1.1(val = 180) 之间的差值。因此，1.k 的值将是 230-180 = 50.
第二个问题是，在某些情况下，我没有子产品类别的数据，但只有聚合级别的数据。与美国出口到欧盟产品类别 3（没有子类别）的情况一样。我想创建一个新的子产品 cat a new，它将产品 cat 的开头与 M（因此 2.M）结合起来，并合并子类别级别中未报告的产品类别级别的值。因此，例如，在美国出口到欧盟产品类别 3 (220) 的情况下，3.M = 220.
如前所述，我认为有两个步骤可以解决我的编码问题。 第一个是关于如何聚合分层数据（请注意，在我的实际数据中，我有 三个 ，而不是两个，子产品级别（例如 1 食品，1.1 水果，1.1。 1 个苹果）。理想情况下，我宁愿避免创建新列，因为我的数据集涉及数百个产品类别。 第二部分是关于执行上述具体操作：1）创建一个具有父节点和子节点之间差异的新类别，2）创建虚拟子节点。 我真的很感谢任何能帮助我的人，因为这是我论文发展的关键。
我知道这是一个复杂的问题，但也欢迎部分回答。
在此先感谢大家的帮助
==============
非常感谢 LAP 的帮助， 这是我在应用函数后遇到的真实数据问题
split2 <- lapply(split1, function(x){
y <- rbind.data.frame(x, x[1,])
y[nrow(y), "Product.cat."] <- paste0(y[nrow(y), "Prodcat2"], "k")
y[nrow(y), "Val"] <- x[1, "Val"] - sum(x[2:nrow(x), "Val"])
return(y)
})
和函数 split3 <- do.call(rbind, split2)
这里是两个splits的head的dput
>dput(Headsplit2)
list(`Algeria.United Arab Emirates.05` = structure(list(Reporter =
c("Algeria",
"Algeria", "Algeria", "Algeria"), Partner = c("United Arab Emirates",
"United Arab Emirates", "United Arab Emirates", "United Arab
Emirates"
), Year = c(2001L, 2001L, 2001L, 2001L), Product.cat. = c("05",
"052", "054", "05k"), `Commodity Description` = c("Fruit and
vegetables",
"Dried fruit including artificially dehydrated", "Vegetables, roots &
tubers, fresh or dried",
"Fruit and vegetables"), `Trade Value` =
structure(c(7.61814641291993e-319,
7.4539189922423e-319, 1.64178014113046e-320, 7.61814641291993e-319
), class = "integer64"), Prodcat1 = c("0", "0", "0", "0"), Prodcat2 =
c("05",
"05", "05", "05")), row.names = c(NA, -4L), vars = c("Reporter",
"Partner", "Prodcat2", "Year"), drop = TRUE, indices = list(0:2),
group_sizes = 3L, biggest_group_size = 3L, labels = structure(list(
Reporter = "Algeria", Partner = "United Arab Emirates", Prodcat2 =
"05",
Year = 2001L), row.names = c(NA, -1L), class = "data.frame", vars =
c("Reporter",
"Partner", "Prodcat2", "Year"), drop = TRUE), class = c("grouped_df",
"tbl_df", "tbl", "data.frame")), `Algeria.United Kingdom.05` =
structure(list(
Reporter = c("Algeria", "Algeria", "Algeria", "Algeria"),
Partner = c("United Kingdom", "United Kingdom", "United Kingdom",
"United Kingdom"), Year = c(2001L, 2001L, 2001L, 2001L),
Product.cat. = c("05", "053", "054", "05k"), `Commodity Description`
= c("Fruit and vegetables",
"Fruit,preserved and fruit preparations", "Vegetables, roots &
tubers, fresh or dried",
"Fruit and vegetables"), `Trade Value` =
structure(c(6.99399328252869e-320,
3.16547859290487e-320, 3.82802062397798e-320, 6.99399328252869e-320
), class = "integer64"), Prodcat1 = c("0", "0", "0", "0"),
Prodcat2 = c("05", "05", "05", "05")), row.names = c(NA,
-4L), vars = c("Reporter", "Partner", "Prodcat2", "Year"), drop =
TRUE, indices = list(
0:2), group_sizes = 3L, biggest_group_size = 3L, labels =
structure(list(
Reporter = "Algeria", Partner = "United Kingdom", Prodcat2 = "05",
Year = 2001L), row.names = c(NA, -1L), class = "data.frame", vars =
c("Reporter",
"Partner", "Prodcat2", "Year"), drop = TRUE), class = c("grouped_df",
"tbl_df", "tbl", "data.frame")), Hungary.Austria.26 = structure(list(
Reporter = c("Hungary", "Hungary", "Hungary", "Hungary",
"Hungary", "Hungary", "Hungary", "Hungary", "Hungary"), Partner =
c("Austria",
"Austria", "Austria", "Austria", "Austria", "Austria", "Austria",
"Austria", "Austria"), Year = c(2000L, 2001L, 2000L, 2000L,
2001L, 2000L, 2000L, 2001L, 2000L), Product.cat. = c("26",
"26", "263", "265", "265", "266", "267", "267", "26k"), `Commodity
Description` = c("Textile fibres, not manufactured, and waste",
"Textile fibres, not manufactured, and waste", "Cotton",
"Vegetable fibres,except cotton and jute", "Vegetable fibres,except
cotton and jute",
"Synthetic and regenerated artificial fibres", "Waste materials from
textile fabrics, incl.rags",
"Waste materials from textile fabrics, incl.rags", "Textile fibres,
not manufactured, and waste"
), `Trade Value` = structure(c(7.3714594359514e-318,
9.95542276370112e-318,
4.94065645841247e-320, 2.96439387504748e-320, 6.91691904177745e-320,
2.32210853545386e-319, 6.33886223614319e-318, 9.60957681161225e-318,
7.3714594359514e-318), class = "integer64"), Prodcat1 = c("2",
"2", "2", "2", "2", "2", "2", "2", "2"), Prodcat2 = c("26",
"26", "26", "26", "26", "26", "26", "26", "26")), row.names = c(NA,
-9L), vars = c("Reporter", "Partner", "Prodcat2", "Year"), drop =
TRUE, indices = list(
c(0L, 2L, 3L, 5L, 6L), c(1L, 4L, 7L)), group_sizes = c(5L,
3L), biggest_group_size = 5L, labels = structure(list(Reporter =
c("Hungary",
"Hungary"), Partner = c("Austria", "Austria"), Prodcat2 = c("26",
"26"), Year = 2000:2001), row.names = c(NA, -2L), class =
"data.frame", vars = c("Reporter",
"Partner", "Prodcat2", "Year"), drop = TRUE), class = c("grouped_df",
"tbl_df", "tbl", "data.frame")), Hungary.Belgium.26 = structure(list(
Reporter = c("Hungary", "Hungary", "Hungary", "Hungary",
"Hungary", "Hungary", "Hungary", "Hungary", "Hungary"), Partner =
c("Belgium",
"Belgium", "Belgium", "Belgium", "Belgium", "Belgium", "Belgium",
"Belgium", "Belgium"), Year = c(2000L, 2001L, 2000L, 2001L,
2000L, 2001L, 2000L, 2001L, 2000L), Product.cat. = c("26",
"26", "265", "265", "266", "266", "267", "267", "26k"), `Commodity
Description` = c("Textile fibres, not manufactured, and waste",
"Textile fibres, not manufactured, and waste", "Vegetable
fibres,except cotton and jute",
"Vegetable fibres,except cotton and jute", "Synthetic and regenerated
artificial fibres",
"Synthetic and regenerated artificial fibres", "Waste materials from
textile fabrics, incl.rags",
"Waste materials from textile fabrics, incl.rags", "Textile fibres,
not manufactured, and waste"
), `Trade Value` = structure(c(3.41893426922143e-318,
7.98410083679454e-318,
3.95252516672997e-320, 9.73309322307256e-319, 1.67488253940183e-318,
1.665001226485e-318, 8.49792910846944e-319, 7.70742407512345e-319,
3.41893426922143e-318), class = "integer64"), Prodcat1 = c("2",
"2", "2", "2", "2", "2", "2", "2", "2"), Prodcat2 = c("26",
"26", "26", "26", "26", "26", "26", "26", "26")), row.names = c(NA,
-9L), vars = c("Reporter", "Partner", "Prodcat2", "Year"), drop =
TRUE, indices = list(
c(0L, 2L, 4L, 6L), c(1L, 3L, 5L, 7L)), group_sizes = c(4L,
4L), biggest_group_size = 4L, labels = structure(list(Reporter =
c("Hungary",
"Hungary"), Partner = c("Belgium", "Belgium"), Prodcat2 = c("26",
"26"), Year = 2000:2001), row.names = c(NA, -2L), class =
"data.frame", vars = c("Reporter",
"Partner", "Prodcat2", "Year"), drop = TRUE), class = c("grouped_df",
"tbl_df", "tbl", "data.frame")), Hungary.Bulgaria.26 =
structure(list(
Reporter = c("Hungary", "Hungary", "Hungary", "Hungary",
"Hungary", "Hungary"), Partner = c("Bulgaria", "Bulgaria",
"Bulgaria", "Bulgaria", "Bulgaria", "Bulgaria"), Year = c(2000L,
2001L, 2000L, 2001L, 2000L, 2000L), Product.cat. = c("26",
"26", "266", "266", "267", "26k"), `Commodity Description` =
c("Textile fibres, not manufactured, and waste",
"Textile fibres, not manufactured, and waste", "Synthetic and
regenerated artificial fibres",
"Synthetic and regenerated artificial fibres", "Waste materials from
textile fabrics, incl.rags",
"Textile fibres, not manufactured, and waste"), `Trade Value` =
structure(c(1.55136612794151e-318,
1.53160350210786e-319, 4.94065645841247e-321, 4.94065645841247e-321,
2.96439387504748e-320, 1.55136612794151e-318), class = "integer64"),
Prodcat1 = c("2", "2", "2", "2", "2", "2"), Prodcat2 = c("26",
"26", "26", "26", "26", "26")), row.names = c(NA, -6L), vars =
c("Reporter",
"Partner", "Prodcat2", "Year"), drop = TRUE, indices = list(c(0L,
2L, 4L), c(1L, 3L)), group_sizes = 3:2, biggest_group_size = 3L,
labels = structure(list(
Reporter = c("Hungary", "Hungary"), Partner = c("Bulgaria",
"Bulgaria"), Prodcat2 = c("26", "26"), Year = 2000:2001), row.names =
c(NA,
-2L), class = "data.frame", vars = c("Reporter", "Partner",
"Prodcat2",
"Year"), drop = TRUE), class = c("grouped_df", "tbl_df", "tbl",
"data.frame")), Hungary.Canada.26 = structure(list(Reporter =
c("Hungary",
"Hungary", "Hungary"), Partner = c("Canada", "Canada", "Canada"
), Year = c(2001L, 2001L, 2001L), Product.cat. = c("26", "265",
"26k"), `Commodity Description` = c("Textile fibres, not
manufactured, and waste",
"Vegetable fibres,except cotton and jute", "Textile fibres, not
manufactured, and waste"
), `Trade Value` = structure(c(8.89318162514244e-320,
6.4228533959362e-320,
8.89318162514244e-320), class = "integer64"), Prodcat1 = c("2",
"2", "2"), Prodcat2 = c("26", "26", "26")), row.names = c(NA,
-3L), vars = c("Reporter", "Partner", "Prodcat2", "Year"), drop =
TRUE, indices = list(
0:1), group_sizes = 2L, biggest_group_size = 2L, labels =
structure(list(
Reporter = "Hungary", Partner = "Canada", Prodcat2 = "26",
Year = 2001L), row.names = c(NA, -1L), class = "data.frame", vars =
c("Reporter",
"Partner", "Prodcat2", "Year"), drop = TRUE), class = c("grouped_df",
"tbl_df", "tbl", "data.frame")))
和拆分 3
dput(Headsplit3)
structure(list(Reporter = c("Algeria", "Algeria", "Algeria",
"Algeria", "Algeria", "Algeria"), Partner = c("United Arab Emirates",
"United Arab Emirates", "United Arab Emirates", "United Arab
Emirates",
"United Kingdom", "United Kingdom"), Year = c(2001L, 2001L, 2001L,
2001L, 2001L, 2001L), Product.cat. = c("05", "052", "054", "05k",
"05", "053"), `Commodity Description` = c("Fruit and vegetables",
"Dried fruit including artificially dehydrated", "Vegetables, roots &
tubers, fresh or dried",
"Fruit and vegetables", "Fruit and vegetables", "Fruit,preserved and
fruit preparations"
), `Trade Value` = structure(c(7.61814641291993e-319,
7.4539189922423e-319,
1.64178014113046e-320, 7.61814641291993e-319, 6.99399328252869e-320,
3.16547859290487e-320), class = "integer64"), Prodcat1 = c("0",
"0", "0", "0", "0", "0"), Prodcat2 = c("05", "05", "05", "05",
"05", "05")), row.names = c(NA, -6L), class = c("grouped_df",
"tbl_df", "tbl", "data.frame"), vars = c("Reporter", "Partner",
"Prodcat2", "Year"), drop = TRUE, indices = list(0:3, 4:5),
group_sizes = c(4L,
2L), biggest_group_size = 4L, labels = structure(list(Reporter =
c("Algeria",
"Algeria"), Partner = c("United Arab Emirates", "United Kingdom"
), Prodcat2 = c("05", "05"), Year = c(2001L, 2001L)), row.names =
c(NA,
-2L), class = "data.frame", vars = c("Reporter", "Partner",
"Prodcat2",
"Year"), drop = TRUE))
如您所见，代码能够识别出阿尔及利亚出口到阿拉伯联合酋长国的 052 和 054 没有将 05 的出口相加（差异仅为 1），并且它确实正确地创建了一个变量05k，但 05k 的交易价值是 154193（= 整个 05 的交易价值）而不是 1。你知道为什么会这样吗？
编辑：好的，我想我明白了！
数据：
df <- data.frame( "Reporter" = c("USA", "USA", "USA", "USA", "USA", "USA","USA", "USA", "USA","USA"),
"Partner" = c( "EU", "EU","EU","EU", "EU","EU","EU", "EU","EU","EU"),
"Product cat." = c("1", "11","111", "12","2", "21", "211", "212", "22", "3"),
"Val" = c(200, 170, 170, 30, 100, 50, 25, 5, 40, 220), stringsAsFactors = FALSE)
我们首先创建两个辅助变量 Prodcat1 和 Prodcat2:
# create new variable Prodcat1
df1 <- df %>% group_by(Reporter, Partner) %>% mutate(Prodcat1 = str_extract(Product.cat., "^.{1}"))
# create new variable Prodcat2 for my 2nd level product category
df1 <- df1 %>% group_by(Reporter, Partner) %>% mutate(Prodcat2 = str_extract(Product.cat., "^.{2}"))
现在我们把数据分成两部分，一部分要完成，一部分不需要第三层的操作：
# to be completed
df2 <- df1 %>%
group_by(Reporter, Partner, Prodcat2) %>%
filter(sum(Val[2:n()]) < Val[1])
# no operation on third level
df3 <- df1 %>%
group_by(Reporter, Partner, Prodcat2) %>%
filter(!sum(Val[2:n()]) < Val[1] | n() == 1)
我们将 df2 拆分为 Prodcat2，控制 Reporter 和 Partner
split1 <- split(df2, interaction(df2$Reporter, df2$Partner, df2$Prodcat2))
split1 <- split1[sapply(split1, nrow) != 0]
并在必要时添加新行：
split2 <- lapply(split1, function(x){
y <- rbind.data.frame(x, x[1,])
y[nrow(y), "Product.cat."] <- paste0(y[nrow(y), "Prodcat2"], "k")
y[nrow(y), "Val"] <- x[1, "Val"] - sum(x[2:nrow(x), "Val"])
return(y)
})
然后我们第一次把数据重新拼起来然后按照原来的Product.cat..
split3 <- do.call(rbind, split2)
newdf <- do.call(rbind, list(split3, df3))
newdf <- newdf %>%
arrange(Product.cat.)
目前数据：
# A tibble: 11 x 6
# Groups: Reporter, Partner, Prodcat2 [5]
Reporter Partner Product.cat. Val Prodcat1 Prodcat2
<chr> <chr> <chr> <dbl> <chr> <chr>
1 USA EU 1 200 1 NA
2 USA EU 11 170 1 11
3 USA EU 111 170 1 11
4 USA EU 12 30 1 12
5 USA EU 2 100 2 NA
6 USA EU 21 50 2 21
7 USA EU 211 25 2 21
8 USA EU 212 5 2 21
9 USA EU 21k 20 2 21
10 USA EU 22 40 2 22
11 USA EU 3 220 3 NA
现在我们进入第二层。首先我们创建三个部分：
# part to complete
df4 <- newdf %>%
group_by(Reporter, Partner, Prodcat1) %>%
filter(nchar(Product.cat.) < 3) %>%
filter(n() == 1 | sum(Val[2:n()]) < Val[1])
# third level rows, which are not necessary here
df5 <- newdf %>%
group_by(Reporter, Partner, Prodcat1) %>%
filter(nchar(Product.cat.) == 3)
# second level part already complete
df6 <- newdf %>%
group_by(Reporter, Partner, Prodcat1) %>%
filter(nchar(Product.cat.) < 3) %>%
filter(sum(Val[2:n()]) == Val[1])
我们再次拆分数据，现在按 Prodcat1，控制 Reporter 和 Partner：
split3 <- split(df4, interaction(df4$Reporter, df4$Partner, df4$Prodcat1))
split3 <- split3[sapply(split3, nrow) != 0]
我们创建新行：
split4 <- lapply(split3, function(x){
if(nrow(x) == 1){
y <- rbind.data.frame(x, x)
y[2, "Product.cat."] <- paste0(y[2, "Prodcat1"], "m")
}else{
y <- rbind.data.frame(x, x[1,])
y[nrow(y), "Product.cat."] <- paste0(y[nrow(y), "Prodcat1"], "k")
y[nrow(y), "Val"] <- x[1, "Val"] - sum(x[2:nrow(x), "Val"])
}
return(y)
})
然后我们将它们重新组合在一起，再次排序并删除辅助变量。
split5 <- do.call(rbind, split4)
finaldf <- do.call(rbind, list(split5, df5, df6))
finaldf <- finaldf %>%
ungroup() %>%
arrange(Product.cat.) %>%
select(-c("Prodcat1", "Prodcat2"))
最终数据：
# A tibble: 13 x 4
Reporter Partner Product.cat. Val
<chr> <chr> <chr> <dbl>
1 USA EU 1 200
2 USA EU 11 170
3 USA EU 111 170
4 USA EU 12 30
5 USA EU 2 100
6 USA EU 21 50
7 USA EU 211 25
8 USA EU 212 5
9 USA EU 21k 20
10 USA EU 22 40
11 USA EU 2k 10
12 USA EU 3 220
13 USA EU 3m 220
最后，我们清除了我们需要的所有临时对象的环境
rm(df1, df2, df3, df4, df5, df6, newdf, split1, split2, split3, split4, split5)
这给我们留下了原始数据集 df 和最终完成的数据集 finaldata :)