Data Science

export HADOOP_HOME=/home/cloud/hadoop-3.3.1
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_OPTS"-Djava.library.path=$HADOOP_HOME/lib/nativ"

append to end

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

in configuration

<property>
     <name>hadoop.tmp.dir</name>
     <value>/home/cloud/tmpdata</value>
     <description>A base for other temporary directories.</description>
 </property>
 <property>
     <name>fs.default.name</name>
     <value>hdfs://localhost:9000</value>
     <description>The name of the default file system></description>
 </property>

in configuration

<property>
  <name>dfs.data.dir</name>
  <value>/home/cloud/dfsdata/namenode</value>
</property>
<property>
  <name>dfs.data.dir</name>
  <value>/home/cloud/dfsdata/datanode</value>
</property>
<property>
  <name>dfs.replication</name>
  <value>1</value>
</property>

in configuration

<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>

<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>
<property>
  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
  <name>yarn.resourcemanager.hostname</name>
  <value>127.0.0.1</value>
</property>
<property>
  <name>yarn.acl.enable</name>
  <value>0</value>
</property>
<property>
  <name>yarn.nodemanager.env-whitelist</name>
  <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PERPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>

cd hadoop-3---
hdfs namenode -format
cd sbin
./start-dfs.sh
./start-yarn.sh
jps

export HADOOP_HOME=/home/cloud/hadoop-3.3.1
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_OPTS"-Djava.library.path=$HADOOP_HOME/lib/nativ"

append to end

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

in configuration

<property>
     <name>hadoop.tmp.dir</name>
     <value>/home/cloud/hadoop-3.3.1/data</value>
     <description>A base for other temporary directories.</description>
 </property>
 <property>
     <name>fs.default.name</name>
     <value>hdfs://hostname:9000</value> watch out for inter floatip for localhost 
     <description>The name of the default file system></description>
 </property>

in configuration

<property>
  <name>dfs.data.dir</name>
  <value>/home/cloud/hahoop-3.3.1/dfsdata/namenode</value>
</property>
<property>
  <name>dfs.data.dir</name>
  <value>/home/cloud/hahoop-3.3.1/dfsdata/datanode</value>
</property>
<property>
  <name>dfs.replication</name>
  <value>3</value>
</property>
<property>
  <name>dfs.namenode.http-address</name>
  <value>*inter floatip:9870*</value>
</property>
<property>
  <name>dfs.namenode.secondary.http-address</name>
  <value>inter floatip:9868</value>
</property>

<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>
<property>
  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
  <name>yarn.resourcemanager.hostname</name>
  <value>*inter floatip*</value>
</property>
<property>
  <name>yarn.acl.enable</name>
  <value>0</value>
</property>
<property>
  <name>yarn.nodemanager.env-whitelist</name>
  <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PERPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>

in configuration

<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>

gwdg01 gwdg10 gwdg19

cd hadoop-3---
xsycn etc/hadoop
hdfs namenode -format
cd sbin
./start-dfs.sh
./start-yarn.sh
jps

this is a full eco system, can build a cluster by my own, with embended scala

my Prof can also build a eco system in pip download file, with config in master: spark-submit –deploy-mode –master yarn test.py But I can't, I can even not find conf file in pip file for pyspark, if you still want to consturcte a cluster, use spark installation from source file, like following

Discrete values {\(x_1\),….,\(x_T\)} = \((x_t)_{t=1}^T\) A core assumpation is the time difference between \(x_{t}\) and \(x_{t+1}\) is equal for \(t \in (1...T)\). \(x_{t}\) can be decomposed into 3 components:

so \[X = T + S + R\]

for \(y=f(x)\) for two points(\(x_1\), \(y_1\)) and (\(x_2, y_2\)), the first-order difference to detrended time series: \(\hat x_{t} = \Delta x^{1}_{t}=x_{t}-x_{t-1}\). or if you want, you can get the second-order-difference \(\Delta^{2}x_{}{t} = \Delta^{1}x_{t}-\Delta^{1}x_{t-1} = x_{t}-2x_{t-1}+x_{t-2}\)

using difference to adjust the seasonal effect: using the difference between two consecutive points in time during the season. \(\hat{\hat{x}} = \Delta_{s} \hat x_{t} = \hat x_{t} - \hat x_{t-s}\)

pro it can deal with both changes in the mean, as well as changes in the movement of the mean

Autocorrelation is the direct relationship of the values of the time series at different points in time, for two adjacent points

Partial autocorrelation is the autocorrelation without the carryover, i.e., only the direct correlation, not for two adjacent points

for Authentication and Partial authentication we can see the residual seasonal effect for regession and seasonal means

three ways to model correlation

1. Definiation:
2. Components separate located
3. communicatation through passing massage between components
4. Characteristics:
5. own memory
6. concurrency
7. locks
8. Applcation:
9. cloud compuation
10. internet of Things
11. Algorithm:

Consensus, Repication

1. Challages:
2. Programm
3. resource sharing

Bit-level, Instruction level, Data level, Task level

1. weather forecast
2. Simulating kernel fusion, tokamak reactor

Stricter than distributed system( strongly scalling: weak scalling)

Recommendation engine

Simulation models real systems to gain new insight Big Data Analytics extracts insight from data

how to deal with big data(5Vs) Raw-> Descriptive -> Diagnostics -> Predictive -> Prescriptive

Raw data, semantic normalization, Data management plan, Data life cycle, data governance, data provenance…

extract from a source database, transform with controlling, error and missing treatment, change the layout to fit loading, integrate them into data warehouses for user

classical: discovery, integration, exploitation in high level, with SQL, java, scala, Python, with parallelism for data Exploration

Lambda architecture is a concept for enabling real-time processing and batch methods together. batch layer(large scala) + serving layer speed layer(read time)

Concurrency, Durability, Consensus,

• relational model
• Clumnar Model (combinded relational model)(HBase)
• key-value model (BigTable)
• Documents model (MongoDB)
• Graph

• an organized collection of data
• software application for user to use the collected data
• a system used for reporting and data analysis, with multidimensional data cube

• Slice
• Dice
• Roll up
• Pivot

Star-Schema: pro: simplification of query and performancd gain, emulates OLAP cube start-Schema: cons: data integrity is not guaranteed, no natural support of many to many relations,

map: filter and convert all input into key-value tuples reduce: receives all tuples with the same keys, accumulated

• HDFS and MapReduce executation engine
• High availability,
• automatic recovery
• Replication of data
• Parallel file access
• Hierarchical namespace
• Rack-awareness

1. distributed code
2. determine fiels
3. map
4. combine
5. shuffle
6. partition
7. reduce
8. output

• Allow modelling and execution of data processing logic
• Reconfigure dataflow graph based on data sizes and target load
• Controlled by vertex management modules
• Task and resource aware scheduling
• Pre-launch and re-use containers and caching intermediate results
• Everyone has to wait for the prozess between mapping and reducing

• Table: Like in relational databases with a schema
• Partitions: table key determining the mapping to directories
• Buckets/Clusters: Data of partitions are mapped into files

• Light-weight index stored within the file
• Compression based on data type
• Concurrent reads of the same file
• Split files without scanning for markers
• Support for adding/removal of fields
• Partial support of table updates
• Partial ACID support (if requested by users)

Identify if an element is a member of a set with n elements Allow false positives but not false negatives

• medium-size object,
• stored by row key,
• cell data is kept in store files on HDFS,
• Encoding can optimize storage space
  • row keys and date
  • column family
  • Reading data

Processing of data stored in memory

• Data will fit in memory
• Additional persistency is required
• Fault-tolerance is mandatory

it based on RDDs, which are immutable tuples, (Resilient Distributed Datasets) Computation is programmed by transformation, lazy evaluation, all computaion is deferred until needed by actions

nums = sc.parallelize(arange(1,100000)) r1 = nums.filter(lambda x: (x%2) == 1) r1 = r1.map(lambda x:(x, x**2)) r1. = r1.reduce(lambda a,b :a * b)

• Transformation: map, filter, union, pipe, groupbykey, join
• Actions: reduct, count, token, frist
• Schuffle: repartation

Application for real-time continuous stream-computation for high-velocity data Stream groupings defines how tuples are transferred

the graph of the calculation represented as network, the parallelism (tasks) is statically defined for a topology

one tuple may be executed multiple time, and if error occurs, tuple restarted from Spout

• each tuple has a tuple ID
• Acker tracks tuple ID with hashing map
• Ack execute each step with XOR of all derived tuple ID, if it retures value 0, retart from Spout agin

• each tuple is executed exactly once,
• provide idempotent operations
• Execute tuples strongly ordered to avoid replicated execution
• Use Storm’s transactional topology(processing phase, commit phase[stong ordering])

Reliable broadcast, Atomic commit, Consensus, Leader election, Replication

Prepare phase, Commit phase

manage the key/value data in distributed system load balancing, and faul tolerant

Consistency(atomicity, visibility, isolation) Availability(Robustness, Scalability, Partition) Durability

CAP(Consistency, Available, Partition tolerance) can't meet together in a DS

Presentation, Application precessing, Data management

Simplicity of the interface, Portability, Cachable, Tracable

Time, cost, energie, Productivity

Computation, Communicatation, Input/Output devices

• Estimate the workload
• Compute the workload throughout per node, W
• Compute the Hardware capabilities P

E = W / P

Color, Size&Shape, Moving,Interpretation of objects,

• show the data
• induce the viewer to think about the substance
• present many numbers in a small space
• make large data sets coherent
• serve a reasonably clear purpose
• be closely integrated with the statistical

• Use the right visualization for data types
• Use building blocks for graphics (known plot styles)
• Reduce information to the essential part to be communicated
• Consistent use of building blocks and themes (retinal properties)

• large data volumes and velocities
• complex system and storage topologies
• understand the system behavior is difficult
• data movement of memory and CPU is costly

• in-situ: analyze results while the applications is still running
• in-transit: analyze data while it is on the IO path
• interact with application while it runs

Can use simple/understandable sequence of operations May use a pattern like a realistic workloads Sometimes only possibility to understand hardware capabilities

Read-ahead, write-behind, async-IO

single-shot: acceptance test periodically: regression test

see in Document folder

mapper

import sys
for line in sys.stdin:
  words = line.strip().split(" ")
    for word in words:
    print(word + "\t" + "1")

reducer

import sys

oldword = ""
count = 0
for line in sys.stdin:
    (word, c) = line.strip().split("\t")
    if word != oldword:
        if count != 0:
            print(oldword +"\t"+ str(count))
        count = 0
        oldword = word
    count = count + int(c)
if oldword != "":
    print(oldword +"\t%d" %(count))

cd /home/hadoop/hadoop-3-3.1/sbin
./start-dfs.sh
./start-yarn.sh
jps

word count example

hdfs daf -put /home/si/Documents/hpda/hpda04-2.3.txt /
hadoop fs -rm -r /hpda04-2.3-output/
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.1.jar wordcount /hpda04-2.3.txt /hpda04-2.3-output/
hadoop fs -cat /hpda04-2.3-output/part-r-00000
cd output
hadoop fs -getmerge /hpda04-2.3-output/ out

With errors

yarn jar share/hadoop/tools/lib/hadoop-streaming-3.3.1.jar -Dmapred.reduce.tasks=1 -Dmapred.map.tasks=11 --mapper /home/si/Documents/hpda/04/mapper.py -reducer /home/si/Documents/hpda/04/reducer.py -input /hpda04-2.3.txt --output /hpda04-2.3-output/

\[ S_{total} = \frac{1}{1-p+\frac{p}{s}} \] \[S = \frac{s}{1-P_{B}-P_{D} + \frac{P_{B}}{N_{B}} + \frac{P_{D}}{N_{D}}} \]

\[ s_{g} = \frac{T_{s} + p T_{p}}{T_{s} + T_{p}} \]

\[ S = \frac{s_g}{(s_g - P_p) + \frac{P_p}{N_p}}\]

all \(P_{p}\) is changed task, such as 70% task doubled, will be 1.4

\[ E = \frac{S}{P}\]

\[ S = \frac{T_{s}}{T_{p}} = \frac{n}{\frac{n}{p}+ \log_{2} p}\]

cpu, interconnection, memory

shared memory distributed memory

easy to build ,hard to large scare

\[T_{parallel} = \frac{T_{serial}}{p}\] speedup: \[S = \frac{T_{serial}}{T_{parallel}}\]. if the parallelized part are perfect parallelable, \(S==p\).

\[T_{parallel} = (1-P)T_{serial} + \frac{P \cdot T_{serial}}{p}\]

speedup: \[S = \frac{T_{serial}}{T_{parallel}} = \frac{1}{(1-P)+ \frac{P}{p}}\]. \[S = \lim_{p -> \infty}\frac{1}{(1-P)+ \frac{P}{p}} = \frac{1}{1-P}\]

\[E = \frac{S}{p}\]

\[ S_{p} = \frac{T_{serial} -p T_{paralle}}{T_{seria} + T_{parallel}}\]

the processes of creating software-based version of resources.

• Utilization: Server consolidation
• Isolation: Implication of errors is restricted in virtual resource only
• Flexiblity: many Application access the same physical Hardware
• On-demand: virtual resource is created/destoryed on request
• Migration: Fault tolerance, live update, optimization of performance
• New reaserch:new OS new technology
• Encapsulation: current stats can be saved copied and loaded
• Minimal downtime
• Fast provisioning

• Translation of instructions
• implantation: Virtual Box
• Hypervisor receive the IO from application,and translate to HW
• Hypervisor translate the request from Guest OS to HW
• no need special HW support
• no need modified OS

• implantation: VMware Workstation
• can install many virtual machine
• need special HW support
• no need modified OS

• VM(modified OS) runs on Host
• Host on hypervisor
• implantation: linux kernel
• need modified OS
• need Host OS level on hypervisor

• Guest OS on Hypervisor
• Hypervisor on Host OS
• Host on HW

• no need modified OS
• need Hypervisor on Host OS
• inter VM communication is difficult

• no hypervisor
• multiple useer instances(light-weight) run on a host OS
• implantation: Docker

• shadow page table on Guest OS
• Extended Page table in Host

hypervisor provide virtual switch, offering every VM a ip address

• Reliablity
• Fast
• small Size
• More expensive
• less Space

• Faster access: because you can have multiple data sources for the same data
• Independence of logic storage resources
• improvement of management: Moving data easy, in multiple localaction
• High reliablity: because of Redundancy
• High effience: Replication and Duplication
• compression, compaction
• increasing volume if needed

• allocate disk space to user on demand
• give a mount of Storage, but not really allocated so much

Single instance Storage: if the hash value of a datablock is the same with one we already stored, dann save its link

• checksum with hash value

compacting the data so that it comsumes less space

Consuming no storage except what is required for metadata until changes are written to the copy

a read-only, point-in-time image of a volume

with more physical disks at the same time

• scaling
• Inreases Repeatablity
• Make processes Faster
• imporve Reliablity
• disadvanage Additional Complexity illusion of Stability

the same code generate the same result, without any change

• Benefits:
  • Repeatablity
  • Agility
  • Disaster Recovery
  • fast deploy
  • live upgrade
• Imperative:describe the stes to get to desired state
• Declarative: describe the desired state

give the initial configuration to run an OS

• Declarative description of resource states
• Client / server Architecture
• Security throgh cettificate
• OS abstraction

collecte data from large mount of servers Watch out the overhitting

• Availability Monitoring: altering of failure
• Capacity Monitoring: detect outages of resource

• Long-term information
• Trend analysis
• Capacity planning

• Measurement: Blackbox, Whitebot,Gauges, Conntes
• Collection: push, pull
• Analysis: real time, short term, long term, Anomaly detection with AI
• Alerting:
• Virtualization

Cloud Computing is a model for enabling on-demand network
access to a shared pool of configurable computing resource
(network, server, storage, application, service) that can
be rapidly provisioned and released with minimal management
effort or service provider interaction

Servive Oriented Architecture SOA has become a core concept of service computing and provides the fundamental technologies for realizing service computing

• No captial costs
• High scalability
• Highh Flexiblity

Different: SDN: software define Network New architectures have a detached control plane instead of heavy logic switching/routing in hardware

• hardware independent
• better shaping and Qos(Quit of service)
• Data Center Briding for local and remote network

• self service front-end
• SSH authenticate
• snapshotting
• using Openstack

• Rapid Time-to-Market
• Minimal Development
• Reduced Pressure on internal resources

based on IaaS, fouce on Applications

• Programmable access
• Distribution over internet
• Encapsulation of discrete functionality
• can offer stardartized Interface
  • TCP/IP prokotoll
  • HTTP based

Simple Object Access Protocol xml based RPC based

Web Services Description Language xml based

• Everything is resource
• Every resource is identified by a unique Identifier
• Using simple and uniform interface
• Communication is done by representation
• be stateless
• more flexiblity
• less redundancy, raw message based
• URI and URL

Application Programming Interface

• Volume: Scale of data
• Velocity : spend of transfer data
• Variety: Different form of data
• Veracity: Uncertainty of data

• Acquisition, Recording
• Extraction, Cleaning, Annotation
• Integration, Representation
• Analysis, Modeling
• Interpretation, Virtualization

• Heterogeneity, Incompleteness
• Scale
• Timeliness
• Privacy

disadvanage: views generated in batch may out of date

disadvanage: expensive and complex

• Apache Storm
• Spark Streaming
• Apache Flink
• Heron

Namenode vs DataNodes

Resource Manager vs NodeManager

• Fast, efficient IO
• Fault tolerant storage
• Publish and Subscribe to steams of records

• Data
• Meta-data
• PID
• Search
• Disposition

A data lake is a data storage, where raw data can be stored, whos structure is determined at the extraction from the lake

Extract transform load

Redundancy for high-availability because of server falied and fast access of data

it is harder to acidentally delete something, such as because of disaster.

ssd

Find-able Accessable Interoperable Reproducible

organizational Operation of server server quality like availability usability

server value value co-creation IT service Management IT service Provider

a framework of best practices of IT service management and delivering

• Development
• Negotiation
• Implementation
• Execution
• Assessment
• Termination

• Parties, terms, conditions
• service defination include costs
• Performance parameters
• what is measured, how and when(monitoring)
• what is done to in case a SLA is voilated

The ability to hide the information from the unauthorized people

The ability to ensure that data are unchanged and remain a correct representation of original data

data is available to authorized people

Meassage: M
Content: N
Ciphertext: C
Public key: E
Encryption: E(x)
private key: D
Decryption: D(x)

RSA Algorithm
1. Select two prime number, p[13] and q[17]
2. Generate Algorithm content N[221]: N = q*p
3. calcalete the Eular function [192]: $\varphi(N)=(p-1)*(q-1)$
4. Rondomly generate public key e[5]: and e is relatively prime with $\varphi(N)$
5. calcalete the private key d[77]: so that $e*d =1$  mod $\varphi(N)$
6. pack Public key E = (n, e) and publish to someone
7. save Private key D =(n, d) 

Someone want to some me Mesaage M: [12]
Encryption: $C = M^{e}$ mod n  [207]
send C [207] to me 

I do the Decryption
Message M: $M=C^{d}$ mod n  [207**77%221]
get the Mesaage [12]

• Integrity
• authentify the sender
• non deniable for message

• challange of key exchange
• en/decryption with the same key

• en/decryption need more resource
• safe key exchange

It's a certificate to identify the sender of message

OS deliver a list of already trusted accepted CAs, it's preconfigured

verifies you are who you say you are

verifies if you have the permission to access data

confusion is to create faint ciphertexts in crytoprahic Diffusion, if one place of plain text the modified, many places can be modified