type
status
slug
summary
tags
category
password
date
icon
Data Science
Data is much easier to access and store
- Prepare students for advanced preparation in machine learning, and statistics.
- Data scientists by providing experience in working with real data, tools and techniques.
- Apply computational and inferential thinking to address real-world problems.
一些好的资源
- python
Teach yourself python. This is a pretty comprehensive tutorial.
python + numpy
python100
- Data Visualization
- Edward Tufte’s book sequences – a classic!
- Prof. Heer’s class.
This short tutorial provides an overview of the basic plotting utilities we will be using.
Altair(Vega-Lite) is a new and powerful visualization library. We might not get to teach it this semester, but you should check it out if you are interested in pursuing visualization deeper. In particular, you should find the example gallery 👇helpful.
This repository contains a series of Python-based Jupyter notebooks that teaches data visualization using Vega-Lite and Altair.
If you are interested in learning more about data visualization, you can find more materials in:
Books
Because data science is a relatively new and rapidly evolving discipline there is no single ideal textbook for this subject. Instead we plan to use reading from a collection of books all of which are free. However, we have listed a few optional books that will provide additional context for those who are interested.
- Principles and Techniques of Data Science, the Data 100 textbook.
- Introduction to Statistical Learning (Free online PDF) This book is a great reference for the machine learning and some of the statistics material in the class
- Data Science from Scratch (Available as eBook for Berkeley students) This more applied book covers many of the topics in this class using Python but doesn’t go into sufficient depth for some of the more mathematical material.
- Doing Data Science (Available as eBook for Berkeley students) This books provides a unique case-study view of data science but uses R and not Python.
- Python for Data Analysis (Available as eBook for Berkeley students). This book provides a good reference for the Pandas library.
- some real-world examples of data science usage are needed.
problem definition→data collection→data pre-proccessing
- how is data collected
- primary data collection
- secondary data collection
web data
- big data
challenges - data volume
- data velocity
- data variety
- data veracity
- the data is stored in the database
- data pre-processing
- data cleansing
- outlier dectection
- Definition and Importance: Outlier detection identifies anomalies in data that may indicate errors, fraudulent behavior, or natural deviations. Detecting outliers can help in identifying system faults, fraud, and errors early, thereby mitigating potential catastrophic consequences (Hodge & Austin, 2004).
- Techniques in High-Dimensional Data: Traditional methods like the Mahalanobis distance are often not applicable in high-dimensional data. New methods using high-breakdown estimators and controlling Type I error have been developed for robust outlier detection in such scenarios (Ro, Zou, Wang, & Yin, 2015).
- Cluster-Based Detection: Outlier detection is not limited to individual points; small clusters exhibiting abnormal behavior in both temporal and spatial dimensions can also be considered outliers. Methods like LDBSCAN are used to detect such cluster-based outliers (Duan, Xu, Liu, & Lee, 2009).
- Bayesian Approaches: Bayesian methods provide a framework for outlier detection in linear models by examining the posterior distribution of random errors. This approach offers a precise definition and methods for exploring data for outliers (Chaloner & Brant, 1988).
- Machine Learning Techniques: Machine learning approaches, such as Generative Adversarial Active Learning, have been employed for unsupervised outlier detection. These methods can generate potential outliers, aiding in the effective separation of outliers from normal data (Liu et al., 2018).
- Distributed and Large-Scale Approaches: Techniques for distributed density-based outlier detection have been developed to handle large-scale data efficiently. These methods compute a Local Outlier Factor (LOF) for each data point, indicating its likelihood of being an outlier (Bai, Wang, Xin, & Wang, 2016).
- Information-Theoretic Methods for Categorical Data: Detecting outliers in categorical data sets can be challenging due to the difficulty in defining a meaningful similarity measure. New approaches using concepts like holoentropy have been proposed to address this challenge (Wu & Wang, 2013).
- data reduction
- data scaling
- data transformation
- 使用编码来解决实际的问题
- 数据清洗：移除数据中的噪声和不相关信息，比如缺失值处理、错误数据修正、重复数据删除等。数据清洗的目的是确保数据的质量和完整性(Hu, 2003)。
- 用户识别和会话识别：在网络日志挖掘中，通过用户识别和会话识别来分析用户行为，这是一个重要的数据预处理步骤(Han Yuan-jie, 2007)。
- 数据集成：合并来自不同来源的数据，解决数据冗余和不一致的问题，以提高数据的可用性和一致性(Liang Wei, 2007)。
- 数据转换和规范化：将数据转换为适合分析的格式，例如规范化、标准化或通过特定的数学函数转换数据(Qian Hong, 2010)。
- 数据离散化：在数据预处理中，数据离散化是将连续属性的值范围减少到一系列值或类别的过程(A. Bakar, Z. Othman, & N. Shuib, 2009)。
- 特征提取和选择：选择对建立模型最有用的特征，以减少数据维度和提高模型的性能(Chithra Y, Prathibha Kiran, & Manoj P B, 2022)。
Outlier detection is a critical process in data analysis, aiming to identify data points that significantly deviate from the majority of the data. These points are known as outliers. The significance of outlier detection spans various fields, including fraud detection, network security, fault detection, and scientific discoveries.
In conclusion, outlier detection is a multifaceted process crucial for identifying anomalies in data sets across various dimensions and applications. It employs a range of statistical and machine learning techniques to effectively identify and handle outliers.
For more AI-powered insights, go to consensus.app.
Z-score标准化（Z-score normalization）是一种常用的数据标准化技术，主要应用于数据预处理过程。这种方法通过调整数据的均值和标准差来实现数据的标准化，使得处理后的数据符合标准正态分布，即均值为0，标准差为1。主要适用于高斯分布的数据中，对于非高斯分布的数据不太实用
总的来说
data pre-process 可以分为下面这些步骤：
数据预处理是数据挖掘和机器学习项目中至关重要的一步。正确的数据预处理步骤可以显著提高数据分析的准确性和效率。以下是一些常见的数据预处理步骤：
总结来说，数据预处理是数据分析和机器学习的基础，涵盖了从数据清洗到特征提取的多个步骤，旨在提高数据质量和分析效率。
- EDA
- 基本原理: EDA的核心是通过直观的方式（如图形和摘要统计）来探索和理解数据集的主要特征。它强调数据的探索性质，而不是预先设定假设进行验证 (Albert & Rizzo, 2012)。
- 方法与技术: EDA包括多种技术，如数据可视化（例如散点图、直方图和箱线图）、统计总结（如均值、中位数和标准差），以及更复杂的数据建模方法 (Gelman, 2004)。
- 应用领域: EDA在许多领域都有应用，包括心理学、医学、社会科学和经济学。它可以帮助研究者在正式的统计分析之前，对数据集有一个初步的理解和认识 (Behrens, 1997)。
- 与确认性数据分析的关系: EDA与确认性数据分析（Confirmatory Data Analysis，CDA）是互补的。EDA帮助形成和精炼假设，而CDA用于测试这些假设。在很多情况下，即使有明确的理论假设，EDA也有助于解释CDA的结果，并可能揭示数据中意想不到或误导性的模式 (Mayer, 1980)。
探索性数据分析（Exploratory Data Analysis，简称EDA）是一种使用统计图形和其他数据可视化方法来分析数据集的过程，旨在揭示数据的基本结构和特征，发现其中的规律性和异常情况。
总结来说，EDA是一种强调数据探索和直观理解的分析方法，它通过图形和统计方法揭示数据的内在结构和特性，对于形成假设和引导后续的深入分析至关重要。
散点图（Scatter Plot）是一种用于数据可视化的常用工具，主要用于探索和展示两个变量之间的关系。以下是散点图的具体用法和实例：
- 基本原理: 散点图通过在坐标轴上绘制点来展示数据集中每个数据点的两个数值变量。这种图表可以帮助用户识别数据之间的相关性、分布模式和异常值 (Keim, Hao, Dayal, Janetzko, & Bak, 2010)。
- 查找相关性: 如果散点图上的点呈现出一定的线性排列趋势，这通常表示两个变量之间存在正相关或负相关。例如，在飞行轨迹分析中，散点图可以用来发现不同航空参数之间的相关性 (Harridon & Rizal, 2020)。
- 识别行为控制: 在应用行为分析中，散点图可以用来识别特定行为的环境控制因素，通过观察行为发生的时间分布模式来确定潜在的环境触发因素 (Touchette, MacDonald, & Langer, 1985)。
- 基因组研究中的应用: 在基因组研究中，散点图用于探索多变量之间的关系，例如，探索不同基因表达水平之间的相关性 (Graffelman & van Eeuwijk, 2005)。
- 优化显示: 对于大量密集的数据点，散点图可能会出现重叠，使得分析变得困难。为此，有研究提出了优化散点图的方法，如使用非均匀的数据分箱技术来减少数据点的重叠 (Fink, Haunert, Spoerhase, & Wolff, 2013)。
总结来说，散点图是一种强大的工具，能够直观地展示两个变量之间的关系和数据分布情况，广泛应用于各类数据分析领域，从基础的相关性探索到复杂的模式识别和行为分析。
- data:anything we learn from
- science:the where, when, how, why, and so what of things
大数据可视化
- dataset into two sets:
- training set (training the dataset)
Digital twins
数字孪生是一个强大的工具，它通过创建物理实体的精确虚拟副本，帮助我们更好地理解和优化现实世界中的各种系统和过程。随着技术的进步，我们可以预见数字孪生在未来将会有更加广泛和深入的应用。
数字孪生是一个物理对象、系统或过程的虚拟模型。这个模型是通过收集大量的数据并利用软件来创建的，它能够精确地反映出其对应的实体在现实世界中的状态和行为。
- what is data-science
- what is data-science not
- data science is not machine learning
- 处理data set的比赛
- 数据科学的处理流程
- 数据形式
- 作者:fufu酱
- 链接:https://csfufu.life/article/87a8cb16-7390-428f-a636-6bb9dfa5df64
- 声明:本文采用 CC BY-NC-SA 4.0 许可协议，转载请注明出处。
相关文章