-
Notifications
You must be signed in to change notification settings - Fork 41
Expand file tree
/
Copy pathCHANGES.txt
More file actions
120 lines (118 loc) · 7.05 KB
/
Copy pathCHANGES.txt
File metadata and controls
120 lines (118 loc) · 7.05 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
/**
* Copyright [2013-2014] PayPal Software Foundation
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
Guagua Change Logs
Changes for Guagua-0.7.7
* Fix a bug on empty gzip files in one worker: remove such woker if size of all gzip file <= 20 bytes.
Changes for Guagua-0.7.6
* Release worker results at early stage to release memory intermediately
Changes for Guagua-0.7.5
* Change message receiving and sending in master/workers to multiple threading.
* Change bug on zookeeper folder created failed exception in client embeded zookeeper.
Changes for Guagua-0.7.4
* Validation data set change on Guagua side: add attachments for GuaguaInputSplit & GuaguaInputFormat
* Fix a bug on embeded ZooKeeper logging enable
* Fix a issue on Kerberos Hadoop: Kerberos is supported in guagua
* Fix Guagua job stablility issue
Changes for Guagua-0.7.3
* Master State Recovery
https://github.com/ShifuML/guagua/issues/94
* Add Combinable Interface to Merge Worker Results When Collecting
https://github.com/ShifuML/guagua/issues/93
Changes for Guagua-0.7.2
* Add GC timer for each iteration.
https://github.com/ShifuML/guagua/issues/90
* Change Embeded zooKeeper default memory from 1G to 512M
https://github.com/ShifuML/guagua/issues/89
* Add completion call back for master and worker
https://github.com/ShifuML/guagua/issues/88
* Auto-retry for bagging failed job.
https://github.com/ShifuML/guagua/issues/77
Changes for Guagua-0.7.1
* Change license from eBay to PayPal
https://github.com/ShifuML/guagua/issues/83
* Fix a bug to let user set 'mapreduce.job.max.split.locations'
https://github.com/ShifuML/guagua/issues/84
Changes for Guagua-0.7.0
* Add workers partial complete feature:
https://github.com/ShifuML/guagua/issues/81
'guagua.min.workers.ratio' by default is 1.0, if set to 0.99, means in each iteration master only wait for 99%
workers. 'guagua.min.workers.timeout': if waiting time over such parameter (default 60000 ms),
'guagua.min.workers.ratio' will be considered.
Changes for Guagua-0.6.1
* Quick fix a bug on Netty Maven dependency:
https://github.com/ShifuML/guagua/issues/76
Changes for Guagua-0.6.0
* Add TCP server/client to send/receive results in both master and workers:
https://github.com/ShifuML/guagua/issues/65.
This issue is to decrease dependency on zookeeper while master results are still stored into zookeeper for fault
tolerance. With this feature, no need to depend on a zookeeper cluster with enough memory and disk. User need not
specify zookeeper server used in shifu and guagua.
* Add memory+(local)disk feature to scale data volume worker can process: https://github.com/ShifuML/guagua/issues/54.
* Add feature to start internal zookeeper server as a process in node of master process:
https://github.com/ShifuML/guagua/issues/14.
* Extract common hadoop classes and APIs to the same packages for in both guagua-mapreduce and guagua-yarn:
https://github.com/ShifuML/guagua/issues/28.
* Embedded zookeeper server(in client node or master node) imporvement:
https://github.com/ShifuML/guagua/issues/62; https://github.com/ShifuML/guagua/issues/61
* Add feature to kill straggling container if over configured threshold in guagua-yarn component:
https://github.com/ShifuML/guagua/issues/52.
Changes for Guagua-0.5.0
* Add annotation ComputableMonitor to support worker time out for each iteration.
* Change worker results in MasterContext from a List to Iteratable to save memory in master.
* Add logistic regression example in guagua-mapreduce-examples.
* Add linear regression example in guagua-mapreduce-examples.
Changes for Guagua-0.4.2
* Add internal(embeded) ZooKeeper support.
* Add default total iteration number if user don't set '-c'.
* Improve readme and other guagua docs.
Changes for Guagua-0.4.1
* Improve guagua shell to follow hadoop command.
* Add kmeans example in guagua-mapreduce-examples.
* Improve API java docs.
Changes for Guagua-0.4.0
* Consistent guagua package in guagua-client. For this package, we have a shell 'guagua' help user select the
platform, user only needs to set the jar, the master and worker classes and other guagua or mapred parameters by
using '-D<key=value>'.
* Support jobs run in Parallel in GuaguaMapReduceClient, only mapreduce adapter support it, yarn adapter doesn't
support it.
Changes for Guagua-0.3.0
* Delete unneeded ZooKeeper znodes ASAP, for example: in iteration 4, delete all znodes from iteration 1 and 2, node
remove znodes for iteration 3 because of fail-over needed.
* Improve Hadoop mapper progress to make sure users can get accurate progress in web and console.
* Improve neural network implementation.
* Fulfill guagua client input validation.
* Add slow task detection mechanism, if one task iteration execution time is over a threshold value, task will kill
itself to trigger fail-over task on other machines.
Changes for Guagua-0.2.0
* Make guagua mapreduce work on Hadoop 0.20.2, 1.2.0 and 2.2.0, test guagua mapreduce on apache, cloudera,
hortonworks hadoop distributions.
* Change default task gc to concurrent gc to decrease gc stopping time to avoid ZooKeeper session timeout.
* Build two packages for guagua-mapreduce to make guagua work on hadoop 1.0 and hadoop yarn.
* Make guagua yarn works, token setting in GuaguaAppMaster.
* Fix the bug: OOM for zookeeper and OOD(disk) for zookeeper.
* Any exception guagua will send kill task signal to Job Tracker to make sure issue task can be killed.
* Add detailed info to mapper status for user to check.
Changes for Guagua-0.1.0
* Master-Workers services implementation for whole guagua arch.
* Sync and async ZooKeeper coordinators for both master and workers.
* MapReduce adapter implementation by using only mapper tasks.
* YARN adapter implementation with fail-over like mapreduce tasks. By default each task has 4 attempts.
* Guagua-core fail-over support to make restarted-task starts at last successful iteration.
* Basic IO for master result and worker results and adapter for hadoop writable.
* Add examples: sum values and distributed neural network implementation.
* Make all intercepters and properties set by hadoop like commnad: '-D<key=value>'.
* Input splits like Pig: set combination size, guagua can combine each worker according to the size.
* Profile intercepter liker memory profilers and worker timers.