-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathcommands.txt
More file actions
58 lines (44 loc) · 3.15 KB
/
Copy pathcommands.txt
File metadata and controls
58 lines (44 loc) · 3.15 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
spark-submit --num-executors 5 --executor-cores 5 --executor-memory 15g \
--conf spark.executorEnv.LD_LIBRARY_PATH=$LD_LIBRARY_PATH \
final_4.py final_op
module load spark/2.4.0
module load python/gnu/3.6.5 gcc/5.3.0
module load anaconda3/2019.10
export PYSPARK_PYTHON=`which python`
-rw-r--r-- 3 htv210 supergroup 2509753777 2020-04-19 15:22 /tmp/bdm/nyc_parking_violation/2015.csv
-rw-r--r-- 3 htv210 supergroup 2066922537 2020-04-19 15:23 /tmp/bdm/nyc_parking_violation/2016.csv
-rw-r--r-- 3 htv210 supergroup 2086913576 2020-04-19 15:23 /tmp/bdm/nyc_parking_violation/2017.csv
-rw-r--r-- 3 htv210 supergroup 2174438972 2020-04-19 15:24 /tmp/bdm/nyc_parking_violation/2018.csv
-rw-r--r-- 3 htv210 supergroup 2003733456 2020-04-19 15:24 /tmp/bdm/nyc_parking_violation/2019.csv
/tmp/bdm/nyc_cscl.csv
unset SSH_ASKPASS
hadoop fs -getmerge BDM_final_output_1000_part_change BDM_output_3.csv
6493.23407292366 - 500 part
7498 - 1000 part
do the groupby,
pivot on the violations after the groupby with columns as years and the values as their
+----------+---------------+---------+---------+---------+---------+---------+---------+
|PHYSICALID|sum(PHYSICALID)|sum(2015)|sum(2016)|sum(2017)|sum(2018)|sum(2019)|OLS_COEFF|
+----------+---------------+---------+---------+---------+---------+---------+---------+
| 3| 108| 824| 590| 752| 839| 345| -70.9|
| 5| 205| 1364| 1402| 1808| 2174| 928| -10.0|
| 6| 228| 2034| 1581| 1625| 1776| 672| -252.9|
| 23| 1679| 9| 5| 163| 83| 314| 68.8|
| 24| 2376| 256| 748| 1812| 3028| 808| 338.4|
| 25| 800| 2| 0| 0| 1| 0| -0.3|
| 45| 540| 12| 9| 42| 21| 3| -0.6|
| 46| 1196| 524| 427| 619| 669| 261| -28.4|
| 47| 1175| 894| 1422| 1867| 1737| 540| -39.3|
| 48| 1824| 2181| 1381| 1730| 1135| 342| -392.4|
| 49| 1519| 1536| 1154| 1186| 1003| 588| -204.7|
| 50| 2400| 11531| 9845| 10399| 8243| 2397| -1987.0|
| 51| 1377| 633| 550| 770| 477| 127| -108.5|
| 52| 4628| 3213| 3285| 3753| 2910| 1170| -446.1|
| 53| 3233| 81| 79| 93| 27| 21| -17.2|
| 56| 4592| 979| 1019| 782| 1266| 687| -33.7|
| 58| 1972| 3| 7| 9| 78| 60| 18.5|
| 59| 826| 11| 36| 97| 60| 30| 6.2|
| 60| 1920| 162| 114| 100| 94| 37| -27.0|
| 61| 915| 3| 0| 3| 0| 0| -0.6|
+----------+---------------+---------+---------+---------+---------+---------+---------+
461.18543219566345