The Experts below are selected from a list of 125985 Experts worldwide ranked by ideXlab platform
Thomas Brox - One of the best experts on this subject based on the ideXlab platform.
-
uncertainty estimates and multi hypotheses networks for Optical Flow
European Conference on Computer Vision, 2018Co-Authors: Eddy Ilg, Ozgun Cicek, Silvio Galesso, Aaron Klein, Osama Makansi, Frank Hutter, Thomas BroxAbstract:Optical Flow estimation can be formulated as an end-to-end supervised learning problem, which yields estimates with a superior accuracy-runtime tradeoff compared to alternative methodology. In this paper, we make such networks estimate their local uncertainty about the correctness of their prediction, which is vital information when building decisions on top of the estimations. For the first time we compare several strategies and techniques to estimate uncertainty in a large-scale computer vision task like Optical Flow estimation. Moreover, we introduce a new network architecture and loss function that enforce complementary hypotheses and provide uncertainty estimates efficiently with a single forward pass and without the need for sampling or ensembles. We demonstrate the quality of the uncertainty estimates, which is clearly above previous confidence measures on Optical Flow and allows for interactive frame rates.
-
uncertainty estimates and multi hypotheses networks for Optical Flow
arXiv: Computer Vision and Pattern Recognition, 2018Co-Authors: Eddy Ilg, Ozgun Cicek, Silvio Galesso, Aaron Klein, Osama Makansi, Frank Hutter, Thomas BroxAbstract:Optical Flow estimation can be formulated as an end-to-end supervised learning problem, which yields estimates with a superior accuracy-runtime tradeoff compared to alternative methodology. In this paper, we make such networks estimate their local uncertainty about the correctness of their prediction, which is vital information when building decisions on top of the estimations. For the first time we compare several strategies and techniques to estimate uncertainty in a large-scale computer vision task like Optical Flow estimation. Moreover, we introduce a new network architecture utilizing the Winner-Takes-All loss and show that this can provide complementary hypotheses and uncertainty estimates efficiently with a single forward pass and without the need for sampling or ensembles. Finally, we demonstrate the quality of the different uncertainty estimates, which is clearly above previous confidence measures on Optical Flow and allows for interactive frame rates.
-
Flownet 2 0 evolution of Optical Flow estimation with deep networks
arXiv: Computer Vision and Pattern Recognition, 2016Co-Authors: Eddy Ilg, Alexey Dosovitskiy, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Thomas BroxAbstract:The FlowNet demonstrated that Optical Flow estimation can be cast as a learning problem. However, the state of the art with regard to the quality of the Flow has still been defined by traditional methods. Particularly on small displacements and real-world data, FlowNet cannot compete with variational methods. In this paper, we advance the concept of end-to-end learning of Optical Flow and make it work really well. The large improvements in quality and speed are caused by three major contributions: first, we focus on the training data and show that the schedule of presenting data during training is very important. Second, we develop a stacked architecture that includes warping of the second image with intermediate Optical Flow. Third, we elaborate on small displacements by introducing a sub-network specializing on small motions. FlowNet 2.0 is only marginally slower than the original FlowNet but decreases the estimation error by more than 50%. It performs on par with state-of-the-art methods, while running at interactive frame rates. Moreover, we present faster variants that allow Optical Flow computation at up to 140fps with accuracy matching the original FlowNet.
-
Flownet learning Optical Flow with convolutional networks
International Conference on Computer Vision, 2015Co-Authors: Alexey Dosovitskiy, Daniel Cremers, Eddy Ilg, Philipp Fischery, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Thomas BroxAbstract:Convolutional neural networks (CNNs) have recently been very successful in a variety of computer vision tasks, especially on those linked to recognition. Optical Flow estimation has not been among the tasks CNNs succeeded at. In this paper we construct CNNs which are capable of solving the Optical Flow estimation problem as a supervised learning task. We propose and compare two architectures: a generic architecture and another one including a layer that correlates feature vectors at different image locations. Since existing ground truth data sets are not sufficiently large to train a CNN, we generate a large synthetic Flying Chairs dataset. We show that networks trained on this unrealistic data still generalize very well to existing datasets such as Sintel and KITTI, achieving competitive accuracy at frame rates of 5 to 10 fps.
-
Flownet learning Optical Flow with convolutional networks
arXiv: Computer Vision and Pattern Recognition, 2015Co-Authors: Philipp Fischer, Daniel Cremers, Eddy Ilg, Alexey Dosovitskiy, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Thomas BroxAbstract:Convolutional neural networks (CNNs) have recently been very successful in a variety of computer vision tasks, especially on those linked to recognition. Optical Flow estimation has not been among the tasks where CNNs were successful. In this paper we construct appropriate CNNs which are capable of solving the Optical Flow estimation problem as a supervised learning task. We propose and compare two architectures: a generic architecture and another one including a layer that correlates feature vectors at different image locations. Since existing ground truth data sets are not sufficiently large to train a CNN, we generate a synthetic Flying Chairs dataset. We show that networks trained on this unrealistic data still generalize very well to existing datasets such as Sintel and KITTI, achieving competitive accuracy at frame rates of 5 to 10 fps.
Michael J. Black - One of the best experts on this subject based on the ideXlab platform.
-
Attacking Optical Flow.
arXiv: Computer Vision and Pattern Recognition, 2019Co-Authors: Anurag Ranjan, Andreas Geiger, Joel Janai, Michael J. BlackAbstract:Deep neural nets achieve state-of-the-art performance on the problem of Optical Flow estimation. Since Optical Flow is used in several safety-critical applications like self-driving cars, it is important to gain insights into the robustness of those techniques. Recently, it has been shown that adversarial attacks easily fool deep neural networks to misclassify objects. The robustness of Optical Flow networks to adversarial attacks, however, has not been studied so far. In this paper, we extend adversarial patch attacks to Optical Flow networks and show that such attacks can compromise their performance. We show that corrupting a small patch of less than 1% of the image size can significantly affect Optical Flow estimates. Our attacks lead to noisy Flow estimates that extend significantly beyond the region of the attack, in many cases even completely erasing the motion of objects in the scene. While networks using an encoder-decoder architecture are very sensitive to these attacks, we found that networks using a spatial pyramid architecture are less affected. We analyse the success and failure of attacking both architectures by visualizing their feature maps and comparing them to classical Optical Flow techniques which are robust to these attacks. We also demonstrate that such attacks are practical by placing a printed pattern into real scenes.
-
unsupervised learning of multi frame Optical Flow with occlusions
European Conference on Computer Vision, 2018Co-Authors: Joel Janai, Michael J. Black, Anurag Ranjan, Fatma Guney, Andreas GeigerAbstract:Learning Optical Flow with neural networks is hampered by the need for obtaining training data with associated ground truth. Unsupervised learning is a promising direction, yet the performance of current unsupervised methods is still limited. In particular, the lack of proper occlusion handling in commonly used data terms constitutes a major source of error. While most Optical Flow methods process pairs of consecutive frames, more advanced occlusion reasoning can be realized when considering multiple frames. In this paper, we propose a framework for unsupervised learning of Optical Flow and occlusions over multiple frames. More specifically, we exploit the minimal configuration of three frames to strengthen the photometric loss and explicitly reason about occlusions. We demonstrate that our multi-frame, occlusion-sensitive formulation outperforms existing unsupervised two-frame methods and even produces results on par with some fully supervised methods.
-
Optical Flow in mostly rigid scenes
Computer Vision and Pattern Recognition, 2017Co-Authors: Jonas Wulff, Laura Sevillalara, Michael J. BlackAbstract:The Optical Flow of natural scenes is a combination of the motion of the observer and the independent motion of objects. Existing algorithms typically focus on either recovering motion and structure under the assumption of a purely static world or Optical Flow for general unconstrained scenes. We combine these approaches in an Optical Flow algorithm that estimates an explicit segmentation of moving objects from appearance and physical constraints. In static regions we take advantage of strong constraints to jointly estimate the camera motion and the 3D structure of the scene over multiple frames. This allows us to also regularize the structure instead of the motion. Our formulation uses a Plane+Parallax framework, which works even under small baselines, and reduces the motion estimation to a one-dimensional search problem, resulting in more accurate estimation. In moving regions the Flow is treated as unconstrained, and computed with an existing Optical Flow method. The resulting Mostly-Rigid Flow (MR-Flow) method achieves state-of-the-art results on both the MPI-Sintel and KITTI-2015 benchmarks.
-
a naturalistic open source movie for Optical Flow evaluation
European Conference on Computer Vision, 2012Co-Authors: Daniel J Butler, Jonas Wulff, Garrett B Stanley, Michael J. BlackAbstract:Ground truth Optical Flow is difficult to measure in real scenes with natural motion. As a result, Optical Flow data sets are restricted in terms of size, complexity, and diversity, making Optical Flow algorithms difficult to train and test on realistic data. We introduce a new Optical Flow data set derived from the open source 3D animated short film Sintel. This data set has important features not present in the popular Middlebury Flow evaluation: long sequences, large motions, specular reflections, motion blur, defocus blur, and atmospheric effects. Because the graphics data that generated the movie is open source, we are able to render scenes under conditions of varying complexity to evaluate where existing Flow algorithms fail. We evaluate several recent Optical Flow algorithms and find that current highly-ranked methods on the Middlebury evaluation have difficulty with this more complex data set suggesting further research on Optical Flow estimation is needed. To validate the use of synthetic data, we compare the image- and Flow-statistics of Sintel to those of real films and videos and show that they are similar. The data set, metrics, and evaluation website are publicly available.
-
A Database and Evaluation Methodology for Optical Flow
International Journal of Computer Vision, 2011Co-Authors: Simon Baker, J. P. Lewis, Daniel Scharstein, Michael J. Black, Stefan Roth, Richard SzeliskiAbstract:The quantitative evaluation of Optical Flow algorithms by Barron et al. ( 1994 ) led to significant advances in performance. The challenges for Optical Flow algorithms today go beyond the datasets and evaluation methods proposed in that paper. Instead, they center on problems associated with complex natural scenes, including nonrigid motion, real sensor noise, and motion discontinuities. We propose a new set of benchmarks and evaluation methods for the next generation of Optical Flow algorithms. To that end, we contribute four types of data to test different aspects of Optical Flow algorithms: (1) sequences with nonrigid motion where the ground-truth Flow is determined by tracking hidden fluorescent texture, (2) realistic synthetic sequences, (3) high frame-rate video used to study interpolation error, and (4) modified stereo sequences of static scenes. In addition to the average angular error used by Barron et al., we compute the absolute Flow endpoint error, measures for frame interpolation error, improved statistics, and results at motion discontinuities and in textureless regions. In October 2007, we published the performance of several well-known methods on a preliminary version of our data to establish the current state of the art. We also made the data freely available on the web at http://vision.middlebury.edu/Flow/ . Subsequently a number of researchers have uploaded their results to our website and published papers using the data. A significant improvement in performance has already been achieved. In this paper we analyze the results obtained to date and draw a large number of conclusions from them.
Luc Van Gool - One of the best experts on this subject based on the ideXlab platform.
-
fast Optical Flow using dense inverse search
European Conference on Computer Vision, 2016Co-Authors: Till Kroeger, Luc Van Gool, Radu Timofte, Dengxin DaiAbstract:Most recent works in Optical Flow extraction focus on the accuracy and neglect the time complexity. However, in real-life visual applications, such as tracking, activity detection and recognition, the time complexity is critical. We propose a solution with very low time complexity and competitive accuracy for the computation of dense Optical Flow. It consists of three parts: (1) inverse search for patch correspondences; (2) dense displacement field creation through patch aggregation along multiple scales; (3) variational refinement. At the core of our Dense Inverse Search-based method (DIS) is the efficient search of correspondences inspired by the inverse compositional image alignment proposed by Baker and Matthews (2001, 2004). DIS is competitive on standard Optical Flow benchmarks. DIS runs at 300 Hz up to 600 Hz on a single CPU core (1024 \(\times \) 436 resolution. 42 Hz/46 Hz when including preprocessing: disk access, image re-scaling, gradient computation. More details in Sect. 3.1.), reaching the temporal resolution of human’s biological vision system. It is order(s) of magnitude faster than state-of-the-art methods in the same range of accuracy, making DIS ideal for real-time applications.
-
fast Optical Flow using dense inverse search
arXiv: Computer Vision and Pattern Recognition, 2016Co-Authors: Till Kroeger, Luc Van Gool, Radu Timofte, Dengxin DaiAbstract:Most recent works in Optical Flow extraction focus on the accuracy and neglect the time complexity. However, in real-life visual applications, such as tracking, activity detection and recognition, the time complexity is critical. We propose a solution with very low time complexity and competitive accuracy for the computation of dense Optical Flow. It consists of three parts: 1) inverse search for patch correspondences; 2) dense displacement field creation through patch aggregation along multiple scales; 3) variational refinement. At the core of our Dense Inverse Search-based method (DIS) is the efficient search of correspondences inspired by the inverse compositional image alignment proposed by Baker and Matthews in 2001. DIS is competitive on standard Optical Flow benchmarks with large displacements. DIS runs at 300Hz up to 600Hz on a single CPU core, reaching the temporal resolution of human's biological vision system. It is order(s) of magnitude faster than state-of-the-art methods in the same range of accuracy, making DIS ideal for visual applications.
-
determination of Optical Flow and its discontinuities using non linear diffusion
European Conference on Computer Vision, 1994Co-Authors: Marc Proesmans, Luc Van Gool, Eric Pauwels, Andre OosterlinckAbstract:A new method for Optical Flow computation by means of a coupled set of non-linear diffusion equations is presented. This approach integrates the classical differential approach with the correlation type of motion detectors. A measure of inconsistency within the Optical Flow field which indicates Optical Flow boundaries. This information is fed back to the Optical Flow equations in a non-linear way and allows the Flow field to be reconstructed while preserving the discontinuities. The whole scheme is also applicable to stereo matching. The model is applied to a set of synthetic and real image sequences to illustrate the behaviour of the coupled diffusion equations.
Eddy Ilg - One of the best experts on this subject based on the ideXlab platform.
-
uncertainty estimates and multi hypotheses networks for Optical Flow
European Conference on Computer Vision, 2018Co-Authors: Eddy Ilg, Ozgun Cicek, Silvio Galesso, Aaron Klein, Osama Makansi, Frank Hutter, Thomas BroxAbstract:Optical Flow estimation can be formulated as an end-to-end supervised learning problem, which yields estimates with a superior accuracy-runtime tradeoff compared to alternative methodology. In this paper, we make such networks estimate their local uncertainty about the correctness of their prediction, which is vital information when building decisions on top of the estimations. For the first time we compare several strategies and techniques to estimate uncertainty in a large-scale computer vision task like Optical Flow estimation. Moreover, we introduce a new network architecture and loss function that enforce complementary hypotheses and provide uncertainty estimates efficiently with a single forward pass and without the need for sampling or ensembles. We demonstrate the quality of the uncertainty estimates, which is clearly above previous confidence measures on Optical Flow and allows for interactive frame rates.
-
uncertainty estimates and multi hypotheses networks for Optical Flow
arXiv: Computer Vision and Pattern Recognition, 2018Co-Authors: Eddy Ilg, Ozgun Cicek, Silvio Galesso, Aaron Klein, Osama Makansi, Frank Hutter, Thomas BroxAbstract:Optical Flow estimation can be formulated as an end-to-end supervised learning problem, which yields estimates with a superior accuracy-runtime tradeoff compared to alternative methodology. In this paper, we make such networks estimate their local uncertainty about the correctness of their prediction, which is vital information when building decisions on top of the estimations. For the first time we compare several strategies and techniques to estimate uncertainty in a large-scale computer vision task like Optical Flow estimation. Moreover, we introduce a new network architecture utilizing the Winner-Takes-All loss and show that this can provide complementary hypotheses and uncertainty estimates efficiently with a single forward pass and without the need for sampling or ensembles. Finally, we demonstrate the quality of the different uncertainty estimates, which is clearly above previous confidence measures on Optical Flow and allows for interactive frame rates.
-
Flownet 2 0 evolution of Optical Flow estimation with deep networks
arXiv: Computer Vision and Pattern Recognition, 2016Co-Authors: Eddy Ilg, Alexey Dosovitskiy, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Thomas BroxAbstract:The FlowNet demonstrated that Optical Flow estimation can be cast as a learning problem. However, the state of the art with regard to the quality of the Flow has still been defined by traditional methods. Particularly on small displacements and real-world data, FlowNet cannot compete with variational methods. In this paper, we advance the concept of end-to-end learning of Optical Flow and make it work really well. The large improvements in quality and speed are caused by three major contributions: first, we focus on the training data and show that the schedule of presenting data during training is very important. Second, we develop a stacked architecture that includes warping of the second image with intermediate Optical Flow. Third, we elaborate on small displacements by introducing a sub-network specializing on small motions. FlowNet 2.0 is only marginally slower than the original FlowNet but decreases the estimation error by more than 50%. It performs on par with state-of-the-art methods, while running at interactive frame rates. Moreover, we present faster variants that allow Optical Flow computation at up to 140fps with accuracy matching the original FlowNet.
-
Flownet learning Optical Flow with convolutional networks
International Conference on Computer Vision, 2015Co-Authors: Alexey Dosovitskiy, Daniel Cremers, Eddy Ilg, Philipp Fischery, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Thomas BroxAbstract:Convolutional neural networks (CNNs) have recently been very successful in a variety of computer vision tasks, especially on those linked to recognition. Optical Flow estimation has not been among the tasks CNNs succeeded at. In this paper we construct CNNs which are capable of solving the Optical Flow estimation problem as a supervised learning task. We propose and compare two architectures: a generic architecture and another one including a layer that correlates feature vectors at different image locations. Since existing ground truth data sets are not sufficiently large to train a CNN, we generate a large synthetic Flying Chairs dataset. We show that networks trained on this unrealistic data still generalize very well to existing datasets such as Sintel and KITTI, achieving competitive accuracy at frame rates of 5 to 10 fps.
-
Flownet learning Optical Flow with convolutional networks
arXiv: Computer Vision and Pattern Recognition, 2015Co-Authors: Philipp Fischer, Daniel Cremers, Eddy Ilg, Alexey Dosovitskiy, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Thomas BroxAbstract:Convolutional neural networks (CNNs) have recently been very successful in a variety of computer vision tasks, especially on those linked to recognition. Optical Flow estimation has not been among the tasks where CNNs were successful. In this paper we construct appropriate CNNs which are capable of solving the Optical Flow estimation problem as a supervised learning task. We propose and compare two architectures: a generic architecture and another one including a layer that correlates feature vectors at different image locations. Since existing ground truth data sets are not sufficiently large to train a CNN, we generate a synthetic Flying Chairs dataset. We show that networks trained on this unrealistic data still generalize very well to existing datasets such as Sintel and KITTI, achieving competitive accuracy at frame rates of 5 to 10 fps.
John L. Barron - One of the best experts on this subject based on the ideXlab platform.
-
The computation of Optical Flow
ACM Computing Surveys, 1995Co-Authors: S. S. Beauchemin, John L. BarronAbstract:Two-dimensional image motion is the projection of the three-dimensional motion of objects, relative to a visual sensor, onto its image plane. Sequences of time-orderedimages allow the estimation of projected two-dimensional image motion as either instantaneous image velocities or discrete image displacements. These are usually called the Optical Flow field or the image velocity field. Provided that Optical Flow is a reliable approximation to two-dimensional image motion, it may then be used to recover the three-dimensional motion of the visual sensor (to within a scale factor) and the three-dimensional surface structure (shape or relative depth) through assumptions concerning the structure of the Optical Flow field, the three-dimensional environment, and the motion of the sensor. Optical Flow may also be used to perform motion detection, object segmentation, time-to-collision and focus of expansion calculations, motion compensated encoding, and stereo disparity measurement. We investigate the computation of Optical Flow in this survey: widely known methods for estimating Optical Flow are classified and examined by scrutinizing the hypothesis and assumptions they use. The survey concludes with a discussion of current research issues.
-
performance of Optical Flow techniques
International Journal of Computer Vision, 1994Co-Authors: John L. Barron, David J Fleet, Steven S BeaucheminAbstract:While different Optical Flow techniques continue to appear, there has been a lack of quantitative evaluation of existing methods. For a common set of real and synthetic image sequences, we report the results of a number of regularly cited Optical Flow techniques, including instances of differential, matching, energy-based, and phase-based methods. Our comparisons are primarily empirical, and concentrate on the accuracy, reliability, and density of the velocity measurements; they show that performance can differ significantly among the techniques we implemented.
