bigdata_analyse 大数据分析项目

Overview

bigdata_analyse

大数据分析项目

wish

采用不同的技术栈,通过对不同行业的数据集进行分析,期望达到以下目标:

  • 了解不同领域的业务分析指标
  • 深化数据处理、数据分析、数据可视化能力
  • 增加大数据批处理、流处理的实践经验
  • 增加数据挖掘的实践经验

tip

  • 项目主要使用的编程语言是 python、sql、hql
  • .ipynb 可以用 jupyter notebook 打开,如何安装, 可以参考 jupyter notebook

jupyter notebook 是一种网页交互形式的 python 编辑器,直接通过 pip 安装,也支持 markdown,很适合用来做数据分析可视化以及写文章、写示例代码等。

list

主题 处理方式 技术栈 数据集下载
1 亿条淘宝用户行为数据分析 离线处理 清洗 hive + 分析 hive + 可视化 echarts 阿里云 或者 百度网盘 提取码:5ipq
1000 万条淘宝用户行为数据实时分析 实时处理 数据源 kafka + 实时分析 flink + 可视化(es + kibana) 百度网盘 提取码:m4mc
300 万条《野蛮时代》的玩家数据分析 离线处理 清洗 pandas + 分析 mysql + 可视化 pyecharts 百度网盘 提取码:paq4
130 万条深圳通刷卡数据分析 离线处理 清洗 pandas + 分析 impala + 可视化 dbeaver 百度网盘 提取码:t561
10 万条厦门招聘数据分析 离线处理 清洗 pandas + 分析 hive + 可视化 ( hue + pyecharts ) + 预测 sklearn 百度网盘 提取码:9wx0
7000 条租房数据分析 离线处理 清洗 pandas + 分析 sqlite + 可视化 matplotlib 百度网盘 提取码:9en3
6000 条倒闭企业数据分析 离线处理 清洗 pandas + 分析 pandas + 可视化 (jupyter notebook + pyecharts) 百度网盘 提取码:xvgm

refer

  1. https://tianchi.aliyun.com/dataset/
  2. https://opendata.sz.gov.cn/data/api/toApiDetails/29200_00403601
  3. https://www.kesci.com/home/dataset
PyIOmica (pyiomica) is a Python package for omics analyses.

PyIOmica (pyiomica) This repository contains PyIOmica, a Python package that provides bioinformatics utilities for analyzing (dynamic) omics datasets.

G. Mias Lab 13 Jun 29, 2022
This is an analysis and prediction project for house prices in King County, USA based on certain features of the house

This is a project for analysis and estimation of House Prices in King County USA The .csv file contains the data of the house and the .ipynb file con

Amit Prakash 1 Jan 21, 2022
Data Analytics: Modeling and Studying data relating to climate change and adoption of electric vehicles

Correlation-Study-Climate-Change-EV-Adoption Data Analytics: Modeling and Studying data relating to climate change and adoption of electric vehicles I

Jonathan Feng 1 Jan 03, 2022
Retentioneering 581 Jan 07, 2023
MapReader: A computer vision pipeline for the semantic exploration of maps at scale

MapReader A computer vision pipeline for the semantic exploration of maps at scale MapReader is an end-to-end computer vision (CV) pipeline designed b

Living with Machines 25 Dec 26, 2022
Catalogue data - A Python Scripts to prepare catalogue data

catalogue_data Scripts to prepare catalogue data. Setup Clone this repo. Install

BigScience Workshop 3 Mar 03, 2022
A Python and R autograding solution

Otter-Grader Otter Grader is a light-weight, modular open-source autograder developed by the Data Science Education Program at UC Berkeley. It is desi

Infrastructure Team 93 Jan 03, 2023
Gaussian processes in TensorFlow

Website | Documentation (release) | Documentation (develop) | Glossary Table of Contents What does GPflow do? Installation Getting Started with GPflow

GPflow 1.7k Jan 06, 2023
Desafio 1 ~ Bantotal

Challenge 01 | Bantotal Please read the instructions for the challenge by selecting your preferred language below: Español Português License Copyright

Maratona Behind the Code 44 Sep 28, 2022
Created covid data pipeline using PySpark and MySQL that collected data stream from API and do some processing and store it into MYSQL database.

Created covid data pipeline using PySpark and MySQL that collected data stream from API and do some processing and store it into MYSQL database.

2 Nov 20, 2021
A lightweight, hub-and-spoke dashboard for multi-account Data Science projects

A lightweight, hub-and-spoke dashboard for cross-account Data Science Projects Introduction Modern Data Science environments often involve many indepe

AWS Samples 3 Oct 30, 2021
An experimental project I'm undertaking for the sole purpose of increasing my Python knowledge

5ePy is an experimental project I'm undertaking for the sole purpose of increasing my Python knowledge. #Goals Goal: Create a working, albeit lightwei

Hayden Covington 1 Nov 24, 2021
CINECA molecular dynamics tutorial set

High Performance Molecular Dynamics Logging into CINECA's computer systems To logon to the M100 system use the following command from an SSH client ss

J. W. Dell 0 Mar 13, 2022
High Dimensional Portfolio Selection with Cardinality Constraints

High-Dimensional Portfolio Selecton with Cardinality Constraints This repo contains code for perform proximal gradient descent to solve sample average

Du Jinhong 2 Mar 22, 2022
Fitting thermodynamic models with pycalphad

ESPEI ESPEI, or Extensible Self-optimizing Phase Equilibria Infrastructure, is a tool for thermodynamic database development within the CALPHAD method

Phases Research Lab 42 Sep 12, 2022
Developed for analyzing the covariance for OrcVIO

about This repo is developed for analyzing the covariance for OrcVIO environment setup platform ubuntu 18.04 using conda conda env create --file envir

Sean 1 Dec 08, 2021
Integrate bus data from a variety of sources (batch processing and real time processing).

Purpose: This is integrate bus data from a variety of sources such as: csv, json api, sensor data ... into Relational Database (batch processing and r

1 Nov 25, 2021
Spectral Analysis in Python

SPECTRUM : Spectral Analysis in Python contributions: Please join https://github.com/cokelaer/spectrum contributors: https://github.com/cokelaer/spect

Thomas Cokelaer 280 Dec 16, 2022
Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoods

Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoods Introduction Graph Neural Networks (GNNs) have demonstrated

37 Dec 15, 2022
BasstatPL is a package for performing different tabulations and calculations for descriptive statistics.

BasstatPL is a package for performing different tabulations and calculations for descriptive statistics. It provides: Frequency table constr

Angel Chavez 1 Oct 31, 2021