Take a list of domains and probe for working HTTP and HTTPS servers

Related tags

Networkinghttprobe
Overview

httprobe

Take a list of domains and probe for working http and https servers.

Install

▶ go get -u github.com/tomnomnom/httprobe

Basic Usage

httprobe accepts line-delimited domains on stdin:

▶ cat recon/example/domains.txt
example.com
example.edu
example.net
▶ cat recon/example/domains.txt | httprobe
http://example.com
http://example.net
http://example.edu
https://example.com
https://example.edu
https://example.net

Extra Probes

By default httprobe checks for HTTP on port 80 and HTTPS on port 443. You can add additional probes with the -p flag by specifying a protocol and port pair:

▶ cat domains.txt | httprobe -p http:81 -p https:8443

Concurrency

You can set the concurrency level with the -c flag:

▶ cat domains.txt | httprobe -c 50

Timeout

You can change the timeout by using the -t flag and specifying a timeout in milliseconds:

▶ cat domains.txt | httprobe -t 20000

Skipping Default Probes

If you don't want to probe for HTTP on port 80 or HTTPS on port 443, you can use the -s flag. You'll need to specify the probes you do want using the -p flag:

▶ cat domains.txt | httprobe -s -p https:8443

Prefer HTTPS

Sometimes you don't care about checking HTTP if HTTPS is working. You can do that with the --prefer-https flag:

▶ cat domains.txt | httprobe --prefer-https

Docker

Build the docker container:

▶ docker build -t httprobe .

Run the container, passing the contents of a file into stdin of the process inside the container. -i is required to correctly map stdin into the container and to the httprobe binary.

▶ cat domains.txt | docker run -i httprobe 
   

   
Comments
  • Install command not work for macOS Mojave

    Install command not work for macOS Mojave

    I re-install go both from official and with brew installer. But there is no difference.

    After command : go get -u github.com/tomnomnom/httprobe execute nothing is change. There is no output in terminal also.

    opened by cyb3rsalih 8
  • Not download httprobe

    Not download httprobe

    Hi,

    i have a problem with the Httprobe because not download..

    sorry for my english XD

    order for install..

    Apt install golang (working done) now go get -u github.com/tomnomnom/httprobe (not error message, but dont download anything..)

    attached screenshots.

    https://ibb.co/c19P9Gb

    https://ibb.co/QQYXKfF

    upload photos from ImgBB.

    opened by densdiego 3
  • Max concurrency?

    Max concurrency?

    Hello tomnomnom!

    I have a short question, does httprobe have a max value for concurrency?

    i am running it against +200k subdomains and sometimes even more, and it is a little bit slow with -c 50 provided. Is there a max value for the -c flag? any other method to speed this up without breaking everything?

    Have a good day.

    opened by marcelo321 2
  • Httprobe installation issue

    Httprobe installation issue

    go version go1.6.2 linux/amd64

    PATH is set as follows:

    export GOPATH=$HOME/go
    export PATH=$PATH:$GOROOT/bin:$GOPATH/bin
    
    ~/go$ go get -u github.com/tomnomnom/httprobe
    # github.com/tomnomnom/httprobe
    src/github.com/tomnomnom/httprobe/main.go:57: unknown http.Transport field 'MaxIdleConns' in struct literal
    src/github.com/tomnomnom/httprobe/main.go:58: unknown http.Transport field 'IdleConnTimeout' in struct literal
    src/github.com/tomnomnom/httprobe/main.go:64: unknown http.Transport field 'DialContext' in struct literal
    src/github.com/tomnomnom/httprobe/main.go:68: undefined: http.ErrUseLastResponse
    
    opened by tacticthreat 2
  • Adding port templates

    Adding port templates

    Currently, if you want to use httprobe and scan additional ports you have to use -p http:<PORT> which is fine for a few ports but for heavy scanning could become tedious.

    I added two port templates to allow scanning for non-standard HTTP ports more easily. The idea is taken from aquatone.

    The templates I implemented are:

    • xlarge for 80, 81, 300, 443, 591, 593, 832, 981, 1010, 1311, 2082, 2087, 2095, 2096, 2480, 3000, 3128, 3333, 4243, 4567, 4711, 4712, 4993, 5000, 5104, 5108, 5800, 6543, 7000, 7396, 7474, 8000, 8001, 8008, 8014, 8042, 8069, 8080, 8081, 8088, 8090, 8091, 8118, 8123, 8172, 8222, 8243, 8280, 8281, 8333, 8443, 8500, 8834, 8880, 8888, 8983, 9000, 9043, 9060, 9080, 9090, 9091, 9200, 9443, 9800, 9981, 12443, 16080, 18091, 18092, 20720, 28017
    • large for 80, 81, 443, 591, 2082, 2087, 2095, 2096, 3000, 8000, 8001, 8008, 8080, 8083, 8443, 8834, 8888

    So if you want to use the xlarge template for example:

    echo api.twitter.com | ./httprobe -p xlarge
    http://api.twitter.com
    https://api.twitter.com
    
    opened by RiccardoAncarani 2
  • Unable to run httprobe from script

    Unable to run httprobe from script

    Hello,

    I have been using httprobe for a while and everything is working fine so I decided to add it to my script, when I run the script I get an httprobe command not found. error. Every other tool in the script ( amass, nmap etc) is working fine and I can still use httprobe perfectly from the terminal in any directory (including the directory the script is in). Any thoughts? Thank you.

    opened by MathematicsUnicorn 1
  • [Feature Request] Additional argument for printing hosts which weren't alive/responded

    [Feature Request] Additional argument for printing hosts which weren't alive/responded

    Hi @tomnomnom,

    Thanks for the amazing tool, It really helps in HTTP probing.

    As mentioned in the title, can we please have an additional argument if a user wants to print out the hosts which didn't respond or were down during the probing? Would help out in domains that point to a STALE record which is terminated etc.

    opened by Anon-Exploiter 1
  • cannot run --prefer-https

    cannot run --prefer-https

    cat abc.txt | httprobe --prefer-https -c 50

    the above command is showing error

    cat abc.txt | httprobe
    #this command works fine though

    I want the result to be either http or https and not both and if https is available I dont want http

    opened by juniorhero 1
  • Read Full Response and Remove Defer Statement

    Read Full Response and Remove Defer Statement

    The isListening function does not reads http bodies which may & do lead to error saying Too many open file descriptors as the connection does not gets closed. Adding an io.Copy to copy the response to ioutil.Discard solves the problem.

    The second thing i did was remove the usage of defer from that code path as calling defer is more expensive than a simple statement.

    opened by Ice3man543 1
  • Automatically figure out whether to prepend http protocol to url or not

    Automatically figure out whether to prepend http protocol to url or not

    Currently, the input url must be in the format:

    google.com
    xyz.com
    

    Pushing code to automatically prepend the HTTP protocol if it is not present, or if its present don't error out, continue with that url.

    All the below URL's are valid input now

    http://google.com
    google.com
    https://google.com
    
    opened by adwaithks 0
  • Better -p Option Formatting

    Better -p Option Formatting

    Hi Tom,

    This pull request addresses the issue that you proposed about specifying a large number of ports being inconvenient given how values passed to the -p option are formatted (#3). The modifications that I made to httprobe allow the following to work:

    $ cat domains.txt | httprobe -p 1234
    $ cat domains.txt | httprobe -p 1234,4321
    $ cat domains.txt | httprobe -p large
    $ cat domains.txt | httprobe -p large,1234
    $ cat domains.txt | httprobe -p http:1234 -p https:1234,4321
    $ cat domains.txt | httprobe -p http:large -p https:xlarge,1234
    

    Determining which ports to probe on each of HTTP and HTTPS now happens only once (when the flags are parsed). Prior to this, httprobe would run a switch case for each host which is a waste of a few CPU cycles (I say this sarcastically :D).

    Also note that I made the relevant changes to the README.md file (including what's mentioned in #23).

    Best Regards, Adam

    opened by NULLHE4D 0
  • add status code filter and follow redirects

    add status code filter and follow redirects

    To avoid getting massive 404 pages on large scopes I added two new flags:

    • -f: Filter the given status code
    • - --follow-redirect Use the default redirection policy, following up to a maximum of 10 times
    opened by itasahobby 0
  • go get: installing executables with 'go get' in module mode is deprecated. 	Use 'go install pkg@version' instead.

    go get: installing executables with 'go get' in module mode is deprecated. Use 'go install [email protected]' instead.

    Help on how to install because even after using the command: go install github.com/tomnomnom/[email protected]

    The is no response from the httprobe in terminal returns: httprobe: command not found

    opened by ZedTechInfo 3
  • Update README.md

    Update README.md

    Installing executables with "go get" in module mode is deprecated. "go install [email protected]" should be used instead. For more information, see https://golang.org/doc/go-get-install-deprecation

    opened by adilsoybali 1
Owner
Tom Hudson
Open-source tool maker, trainer, talker, fixer, eater, not really a sheep. He/him.
Tom Hudson
A python tool auto change proxy or ip after dealy time set by user

Auto proxy Ghost This tool auto change proxy or ip after dealy time set by user how to run 1. Install required file ./requirements.sh 2.Enter command

Harsh Tagra 0 Feb 23, 2022
jarbou3 is rat tool coded in python with C&C which can accept multiple connections from clients

jarbou3 Jarbou3 is rat tool with coded in python with C&C which can accept multi

youhacker55 108 Dec 29, 2022
ProtOSINT is a Python script that helps you investigate Protonmail accounts and ProtonVPN IP addresses

ProtOSINT ProtOSINT is a Python script that helps you investigate ProtonMail accounts and ProtonVPN IP addresses. Description This tool can help you i

pixelbubble 249 Dec 23, 2022
An ansible playbook to set up wireguard server.

Poor man's VPN (pay for only what you need) An ansible playbook to quickly set up Wireguard server for occasional personal use. It takes around five m

Amrit Bera 613 Dec 25, 2022
BibleNotifyDesktop - Desktop version of Bible Notify

Bible Notify Desktop This is the repository for the Desktop version of the daily

Bible Notify 5 Nov 16, 2022
Enrich IP addresses with metadata and security IoC

Stratosphere IP enrich Get an IP address and enrich it with metadata and IoC You need API keys for VirusTotal and PassiveTotal (RiskIQ) How to use fro

Stratosphere IPS 10 Sep 25, 2022
Utility for converting IP Fabric webhooks into a Teams format.

IP Fabric Webhook Integration for Microsoft Teams Setup IP Fabric Setup Go to Settings Webhooks Add webhook Provide a name URL will be: 'http://Y

Community Fabric 1 Jan 26, 2022
A live streaming chatroom involving multiple modalities, such as voice, gesture, and facial expression

HiLive A live streaming chatroom involving multiple modalities, such as voice, gesture, and facial expression. Introduction We focus on demonstrating

Ryan Yen 2 Dec 02, 2021
An automatic reaction network generator for reactive molecular dynamics simulation.

ReacNetGenerator An automatic reaction network generator for reactive molecular dynamics simulation. ReacNetGenerator: an automatic reaction network g

Tong Zhu Research Group 35 Dec 14, 2022
This is a python based command line Network Scanner utility, which input as an argument for the exact IP address or the relative IP Address range you wish to do the Network Scan for and returns all the available IP addresses with their MAC addresses on your current Network.

This is a python based command line Network Scanner utility, which input as an argument for the exact IP address or the relative IP Address range you wish to do the Network Scan for and returns all t

Abhinandan Khurana 1 Feb 09, 2022
Fmog: Fortinet Mass Object Generator. This script will take a list of IP addresses and create address objects with the same name

Fmog: Fortinet Mass Object Generator This script will take a list of IP addresses and create address objects with the same name. It will also add them

2 Oct 26, 2021
sync application configuration and settings across multiple multiplatform devices

sync application configuration and settings across multiple multiplatform devices ✨ Key Features • ⚗️ Installation • 📑 How To Use • 🤔 FAQ • 🛠️ Setu

Souvik 6 Aug 25, 2022
FramIp - it a framework for work at IP and domain

FramIp FramIp - it a framework for work with IP and domain Installation (termux) $ pkg install git && pkg install python && git clone https://github.c

1 Jan 28, 2022
Compare the contents of your hosted and proxy repositories for coordinate collisions

Nexus Repository Manager dependency/namespace confusion checker This repository contains a script to check if you have artifacts containing the same n

Sonatype Community 59 Mar 31, 2022
A simple python application for generating a WiFi QR code for ease of connection

A simple python application for generating a WiFi QR code Initialize the class by providing QR code values WiFi_QR_Code(self, error_correction: int =

Ivan 2 Aug 01, 2022
Dokumentasi belajar Network automation

Repositori belajar network automation dengan Docker, Python & GNS3 Using Frameworks and integrate with: Paramiko Netmiko Telnetlib CSV SFTP Netmiko, S

Daniel.Pepuho 3 Mar 15, 2022
A Python based command line ARP Spoofer utility, which takes input as arguments for the exact target IP and gateway IP for which you wish to Spoof ARP request

A Python based command line ARP Spoofer utility, which takes input as arguments for the exact target IP and gateway IP for which you wish to Spoof ARP request

Abhinandan Khurana 1 Feb 10, 2022
RabbitMQ asynchronous connector library for Python with built in RPC support

About RabbitMQ connector library for Python that is fully integrated with the aio-pika framework. Introduction BunnyStorm is here to simplify working

22 Sep 11, 2022
A gRPC-Web implementation for Python

Sonora Sonora is a Python-first implementation of gRPC-Web built on top of standard Python APIs like WSGI and ASGI for easy integration. Why? Regular

Alex Stapleton 216 Dec 30, 2022
snappi-trex is a snappi plugin that allows executing scripts written using snappi with Cisco's TRex Traffic Generator

snappi-trex snappi-trex is a snappi plugin that allows executing scripts written using snappi with Cisco's TRex Traffic Generator Design snappi-trex c

Open Traffic Generator 14 Sep 07, 2022