The Experts below are selected from a list of 69 Experts worldwide ranked by ideXlab platform
Ilan Shomorony - One of the best experts on this subject based on the ideXlab platform.
-
spectral jaccard similarity a new approach to estimating pairwise sequence alignments
Patterns, 2020Co-Authors: Tavor Z Baharav, Govinda M Kamath, Ilan ShomoronyAbstract:Summary Pairwise sequence alignment is often a computational bottleneck in genomic analysis pipelines, particularly in the context of third-generation sequencing technologies. To speed up this process, the pairwise k-mer Jaccard similarity is sometimes used as a proxy for alignment size in order to filter pairs of reads, and min-Hashes are employed to efficiently estimate these similarities. However, when the k-mer distribution of a dataset is significantly non-uniform (e.g., due to GC biases and repeats), Jaccard similarity is no longer a good proxy for alignment size. In this work, we introduce a min-Hash-based approach for estimating alignment sizes called Spectral Jaccard Similarity, which naturally accounts for uneven k-mer distributions. The Spectral Jaccard Similarity is computed by performing a singular value decomposition on a min-Hash Collision matrix. We empirically show that this new metric provides significantly better estimates for alignment sizes, and we provide a computationally efficient estimator for these spectral similarity scores.
-
spectral jaccard similarity a new approach to estimating pairwise sequence alignments
bioRxiv, 2019Co-Authors: Tavor Z Baharav, Govinda M Kamath, Ilan ShomoronyAbstract:A key step in many genomic analysis pipelines is the identification of regions of similarity between pairs of DNA sequencing reads. This task, known as pairwise sequence alignment, is a heavy computational burden, particularly in the context of third-generation long-read sequencing technologies, which produce noisy reads. This issue is commonly addressed via a two-step approach: first, we filter pairs of reads which are likely to have a large alignment, and then we perform computationally intensive alignment algorithms only on the selected pairs. The Jaccard similarity between the set of k -mers of each read can be shown to be a proxy for the alignment size, and is usually used as the filter. This strategy has the added benefit that the Jaccard similarities don’t need to be computed exactly, and can instead be efficiently estimated through the use of min-Hashes . This is done by Hashing all k -mers of a read and computing the minimum Hash value (the min-Hash) for each read. For a randomly chosen Hash function, the probability that the min-Hashes are the same for two distinct reads is precisely their k -mer Jaccard similarity. Hence, one can estimate the Jaccard similarity by computing the fraction of min-Hash Collisions out of the set of Hash functions considered. However, when the k -mer distribution of the reads being considered is significantly non-uniform, Jaccard similarity is no longer a good proxy for the alignment size. In particular, genome-wide GC biases and the presence of common k -mers increase the probability of a min-Hash Collision, thus biasing the estimate of alignment size provided by the Jaccard similarity. In this work, we introduce a min-Hash-based approach for estimating alignment sizes called Spectral Jaccard Similarity which naturally accounts for an uneven k -mer distribution in the reads being compared. The Spectral Jaccard Similarity is computed by considering a min-Hash Collision matrix (where rows correspond to pairs of reads and columns correspond to different Hash functions), removing an offset, and performing a singular value decomposition . The leading left singular vector provides the Spectral Jaccard Similarity for each pair of reads. In addition, we develop an approximation to the Spectral Jaccard Similarity that can be computed with a single matrix-vector product, instead of a full singular value decomposition. We demonstrate improvements in AUC of the Spectral Jaccard Similarity based filters over Jaccard Similarity based filters on 40 datasets of PacBio reads from the NCTC collection. The code is available at [https://github.com/TavorB/spectral\_jaccard\_similarity][1]. [1]: https://github.com/TavorB/spectral_jaccard_similarity
Wang Zhi - One of the best experts on this subject based on the ideXlab platform.
-
DeepC2: AI-powered Covert Botnet Command and Control on OSNs
2021Co-Authors: Wang Zhi, Liu Chaoge, Cui Xiang, Zhang Jialong, Yin Jie, Liu Jiaxi, Tian ZhihongAbstract:Botnets are one of the major threats to computer security. In previous botnet command and control (C&C) scenarios using online social networks (OSNs), methods for addressing (e.g., IDs, links, or DGAs) are hardcoded into bots. Once a bot is reverse engineered, the botmaster and C&C infrastructure will be exposed. Additionally, abnormal content from explicit commands may expose botmasters and raise anomalies on OSNs. To overcome these deficiencies, we propose DeepC2, an AI-powered covert C&C method on OSNs. By leveraging neural networks, bots can find botmasters by avatars, which are converted into feature vectors and embedded into bots. Adversaries cannot derive botmasters' accounts from the vectors. Commands are embedded into normal contents (e.g., tweets and comments) using text data augmentation and Hash Collision. Experiments on Twitter show that command-embedded contents can be generated efficiently, and bots can find botmasters and obtain commands accurately. Security analysis on different scenarios show that DeepC2 is robust and hard to be shut down. By demonstrating how AI may help promote covert communication on OSNs, this work provides a new perspective on botnet detection and confrontation.Comment: 12 pages, 15 figures, 7 tables. Security analysis and discussion update
-
AI-powered Covert Botnet Command and Control on OSNs
2020Co-Authors: Wang Zhi, Liu Chaoge, Cui Xiang, Zhang Jialong, Yin Jie, Liu Jiaxi, Liu Qixu, Zhang JinliAbstract:Botnet is one of the major threats to computer security. In previous botnet command and control (C&C) scenarios using online social networks (OSNs), methods for finding botmasters (e.g. ids, links, DGAs, etc.) are hardcoded into bots. Once a bot is reverse engineered, botmaster is exposed. Meanwhile, abnormal contents from explicit commands may expose botmaster and raise anomalies on OSNs. To overcome these deficiencies, we propose an AI-powered covert C&C channel. On leverage of neural networks, bots can find botmasters by avatars, which are converted into feature vectors. Commands are embedded into normal contents (e.g. tweets, comments, etc.) using text data augmentation and Hash Collision. Experiment on Twitter shows that the command-embedded contents can be generated efficiently, and bots can find botmaster and obtain commands accurately. By demonstrating how AI may help promote a covert communication on OSNs, this work provides a new perspective on botnet detection and confrontation.Comment: 12 pages, 14 figures, 8 tables; an experiment added; discussions update
Tavor Z Baharav - One of the best experts on this subject based on the ideXlab platform.
-
spectral jaccard similarity a new approach to estimating pairwise sequence alignments
Patterns, 2020Co-Authors: Tavor Z Baharav, Govinda M Kamath, Ilan ShomoronyAbstract:Summary Pairwise sequence alignment is often a computational bottleneck in genomic analysis pipelines, particularly in the context of third-generation sequencing technologies. To speed up this process, the pairwise k-mer Jaccard similarity is sometimes used as a proxy for alignment size in order to filter pairs of reads, and min-Hashes are employed to efficiently estimate these similarities. However, when the k-mer distribution of a dataset is significantly non-uniform (e.g., due to GC biases and repeats), Jaccard similarity is no longer a good proxy for alignment size. In this work, we introduce a min-Hash-based approach for estimating alignment sizes called Spectral Jaccard Similarity, which naturally accounts for uneven k-mer distributions. The Spectral Jaccard Similarity is computed by performing a singular value decomposition on a min-Hash Collision matrix. We empirically show that this new metric provides significantly better estimates for alignment sizes, and we provide a computationally efficient estimator for these spectral similarity scores.
-
spectral jaccard similarity a new approach to estimating pairwise sequence alignments
bioRxiv, 2019Co-Authors: Tavor Z Baharav, Govinda M Kamath, Ilan ShomoronyAbstract:A key step in many genomic analysis pipelines is the identification of regions of similarity between pairs of DNA sequencing reads. This task, known as pairwise sequence alignment, is a heavy computational burden, particularly in the context of third-generation long-read sequencing technologies, which produce noisy reads. This issue is commonly addressed via a two-step approach: first, we filter pairs of reads which are likely to have a large alignment, and then we perform computationally intensive alignment algorithms only on the selected pairs. The Jaccard similarity between the set of k -mers of each read can be shown to be a proxy for the alignment size, and is usually used as the filter. This strategy has the added benefit that the Jaccard similarities don’t need to be computed exactly, and can instead be efficiently estimated through the use of min-Hashes . This is done by Hashing all k -mers of a read and computing the minimum Hash value (the min-Hash) for each read. For a randomly chosen Hash function, the probability that the min-Hashes are the same for two distinct reads is precisely their k -mer Jaccard similarity. Hence, one can estimate the Jaccard similarity by computing the fraction of min-Hash Collisions out of the set of Hash functions considered. However, when the k -mer distribution of the reads being considered is significantly non-uniform, Jaccard similarity is no longer a good proxy for the alignment size. In particular, genome-wide GC biases and the presence of common k -mers increase the probability of a min-Hash Collision, thus biasing the estimate of alignment size provided by the Jaccard similarity. In this work, we introduce a min-Hash-based approach for estimating alignment sizes called Spectral Jaccard Similarity which naturally accounts for an uneven k -mer distribution in the reads being compared. The Spectral Jaccard Similarity is computed by considering a min-Hash Collision matrix (where rows correspond to pairs of reads and columns correspond to different Hash functions), removing an offset, and performing a singular value decomposition . The leading left singular vector provides the Spectral Jaccard Similarity for each pair of reads. In addition, we develop an approximation to the Spectral Jaccard Similarity that can be computed with a single matrix-vector product, instead of a full singular value decomposition. We demonstrate improvements in AUC of the Spectral Jaccard Similarity based filters over Jaccard Similarity based filters on 40 datasets of PacBio reads from the NCTC collection. The code is available at [https://github.com/TavorB/spectral\_jaccard\_similarity][1]. [1]: https://github.com/TavorB/spectral_jaccard_similarity
Zhang Jinli - One of the best experts on this subject based on the ideXlab platform.
-
AI-powered Covert Botnet Command and Control on OSNs
2020Co-Authors: Wang Zhi, Liu Chaoge, Cui Xiang, Zhang Jialong, Yin Jie, Liu Jiaxi, Liu Qixu, Zhang JinliAbstract:Botnet is one of the major threats to computer security. In previous botnet command and control (C&C) scenarios using online social networks (OSNs), methods for finding botmasters (e.g. ids, links, DGAs, etc.) are hardcoded into bots. Once a bot is reverse engineered, botmaster is exposed. Meanwhile, abnormal contents from explicit commands may expose botmaster and raise anomalies on OSNs. To overcome these deficiencies, we propose an AI-powered covert C&C channel. On leverage of neural networks, bots can find botmasters by avatars, which are converted into feature vectors. Commands are embedded into normal contents (e.g. tweets, comments, etc.) using text data augmentation and Hash Collision. Experiment on Twitter shows that the command-embedded contents can be generated efficiently, and bots can find botmaster and obtain commands accurately. By demonstrating how AI may help promote a covert communication on OSNs, this work provides a new perspective on botnet detection and confrontation.Comment: 12 pages, 14 figures, 8 tables; an experiment added; discussions update
Mounira Kourjieh - One of the best experts on this subject based on the ideXlab platform.
-
a symbolic intruder model for Hash Collision attacks
ASIAN'06 Proceedings of the 11th Asian computing science conference on Advances in computer science: secure software and related issues, 2006Co-Authors: Yannick Chevalier, Mounira KourjiehAbstract:In the recent years, several practical methods have been published to compute Collisions on some commonly used Hash functions. Starting from two messages m1 and m2 these methods permit to compute m′1 and m′2 similar to the former such that they have the same image for a given Hash function. In this paper we present a method to take into account, at the symbolic level, that an intruder actively attacking a protocol execution may use these Collision algorithms in reasonable time during the attack. This decision procedure relies on the reduction of constraint solving for an intruder exploiting the Collision properties of Hash functions to constraint solving for an intruder operating on words, that is with an associative symbol of concatenation. The decidability of the latter is interesting in its own right as it is the first decidability result that we are aware of for an intruder system for which unification is infinitary, and permits to consider in other contexts an associative concatenation of messages instead of their pairing.
-
a symbolic intruder model for Hash Collision attacks
CSTVA'06, 2006Co-Authors: Yannick Chevalier, Mounira KourjiehAbstract:In the recent years, several practical methods have been published to compute Collisions on some commonly used Hash functions. In this paper we present a method to take into account, at the symbolic level, that an intruder actively attacking a protocol execution may use these Collision algorithms in reasonable time during the attack. Our decision procedure relies on the reduction of constraint solving for an intruder exploiting the Collision properties of hush functions to constraint solving for an intruder operating on words.
